:W&Bá¾GyÅ”M n20ypkh2ª; 9 /0.29.0+7872fd9d9f86ca5c50617e19cb168122b7f959480.70.0'LQäû‚" x8gf0s4d9xea n20ypkh2Š× n20ypkh2juju11-uc-san-diegoSPAtop"š _wandb‚{}  batch_size‚256  dataset_limit‚0.2  dropout‚0.19293077014535664 epochs‚10  hidden_dim‚128 ' learning_rate‚0.0008577006004054611  num_attention_heads‚8  num_branch_encoder_layers‚7  num_embedding_layers‚14  num_encoder_layers‚4Bjjtttt_blocks_n20ypkh2bs2s2ev0zj*jj-spatop-sweepagent-tttt-pairwise-1-8pwgvŠ ¥êÔÀ—͵š3 (H¨¸hp"3.12.10*0.29.0j linux-x86_64 Qs7 n20ypkh2&  _wandb.runtime‚0  _runtime‚0¾ÐÕEjÒä -Linux-6.8.0-137-generic-x86_64-with-glibc2.36CPython 3.12.10 ¤êÔ°ûæ‹*-of*=/data/spatop/options_files/sweep_tttt_jj/config_n20ypkh2.json*-l* /scratch/logs*-n*jjtttt_blocks_n20ypkh22-m spanet.trainRjacksonjulian2014@gmail.comZ /scratch/logsb*jj-spatop-sweepagent-tttt-pairwise-1-8pwgvr/opt/conda/bin/pythonº yn4zqigak2xm7dn828un8fdlzxlyrydwæPò2  requirements.txtU^¨sW n20ypkh2ZF (H¨¸ (H¨¸ (hpð"3.12.10*0.29.0j linux-x86_64ÚD`Y n20ypkh2ZH (H¨¸ (H¨¸ (8hpð"3.12.10*0.29.0j linux-x86_64Ö)Æ* n20ypkh2b trainer/global_step29WtÙ\  n20ypkh2ZK (H¨¸ (H¨¸(8hpð"3.12.10*0.29.0j linux-x86_64Åúp@/  n20ypkh2b*"trainer/global_step2BɪK  n20ypkh2j: ¦êÔ°¸í¡*LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] U?fÉ ÒÀ€ˆ€’NVIDIA A100-SXM4-80GB˜¢ /€€€øáo€à¹Ü×Tª€àäêÞÂS NVIDIA A100-SXM4-80GB€€€€À€6"Ampere*(GPU-c1fdb213-5a16-84e3-13b4-cf29bc35dbf8Ê13.2º yn4zqigak2xm7dn828un8fdlzxlyrydwL’  n20ypkh2j€  §êÔ€þ¸ î â”â”â”â”┳â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”┳â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”┳â”â”â”â”â”â”â”â”┳â”â”â”â”â”â”â”┳â”â”â”â”â”â”â”┓ ┃ ┃ Name ┃ Type ┃ Params ┃ Mode ┃ FLOPs ┃ ┡â”â”â”╇â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”╇â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”╇â”â”â”â”â”â”â”â”╇â”â”â”â”â”â”â”╇â”â”â”â”â”â”â”┩ │ 0 │ embedding │ MultiInputVectorEmbed… │ 866 K │ train │ 0 │ │ 1 │ encoder │ JetEncoderWithPairwise │ 1.3 M │ train │ 0 │ │ 2 │ branch_decoders │ ModuleList │ 53.9 M │ train │ 0 │ │ 3 │ regression_decoder │ RegressionDecoder │ 0 │ train │ 0 │ │ 4 │ classification_decoder │ ClassificationDecoder │ 0 │ train │ 0 │ │ │ other params │ n/a │ 48 │ n/a │ n/a │ └───┴────────────────────────┴────────────────────────┴────────┴───────┴───────┘ [Êq[ n20ypkh2jÉ §êÔàê± ·Trainable params: 56.1 M Non-trainable params: 86 Total params: 56.1 M Total estimated model params size (MB): 224.267 Modules in train mode: 5866 Modules in eval mode: 0 Total FLOPs: 0 r_7np n20ypkh2j_ §êÔðíÊ Pwandb: WARNING Config item 'hidden_dim' was locked by 'sweep' (ignored update). %I^ z n20ypkh2ji §êÔ¸ÇÏ Zwandb: WARNING Config item 'num_embedding_layers' was locked by 'sweep' (ignored update). 6’Ç¥x n20ypkh2jg §êÔÐ³Ó Xwandb: WARNING Config item 'num_encoder_layers' was locked by 'sweep' (ignored update). )$Å- n20ypkh2jn §êÔ˜× _wandb: WARNING Config item 'num_branch_encoder_layers' was locked by 'sweep' (ignored update). Ûi¦›y n20ypkh2jh §êÔ¨ÕÚ Ywandb: WARNING Config item 'num_attention_heads' was locked by 'sweep' (ignored update). @Eës n20ypkh2jb §êÔ€åç Swandb: WARNING Config item 'dataset_limit' was locked by 'sweep' (ignored update). ä:p n20ypkh2j_ §êÔ¡î Pwandb: WARNING Config item 'batch_size' was locked by 'sweep' (ignored update). |‘…žs n20ypkh2jb §êÔ‰ö Swandb: WARNING Config item 'learning_rate' was locked by 'sweep' (ignored update). àJRm n20ypkh2j\ §êÔØÖþ Mwandb: WARNING Config item 'dropout' was locked by 'sweep' (ignored update). dÂw"l n20ypkh2j[ §êÔ¸¢… Lwandb: WARNING Config item 'epochs' was locked by 'sweep' (ignored update). ãp[×Ü n20ypkh2*Ê  transformer_dim‚128  transformer_dim_scale‚2.0  initial_embedding_dim‚128  position_embedding_dim‚128 ! num_branch_embedding_layers‚3  num_jet_embedding_layers‚0  num_jet_encoder_layers‚2  num_detector_layers‚2  num_regression_layers‚3  num_classification_layers‚3  split_symmetric_attention‚1 ! transformer_activation‚"gelu"  skip_connections‚1 ( "initial_embedding_skip_connections‚1  linear_block_type‚"GRU"  transformer_type‚"Gated"  linear_activation‚"gelu"  normalization‚ "LayerNorm"  masking‚ "Filling" linear_prelu_activation‚true  use_pairwise_interactions‚1  num_pairwise_features‚4  pairwise_embedding_dim‚8  pairwise_input_source‚"all"  pairwise_block_embeddings‚1  pairwise_cross_type‚true  pairwise_feature_set‚""  pairwise_eq2to2‚false  use_moe‚false  num_experts‚8  num_experts_per_tok‚2  moe_loss_scale‚0.01 F event_info_file‚2"/data/spatop/event_files/tttt/tttt_hadronic.yaml" = training_file‚+"/data/spatop/tttt_15M/h5/tttt_training.h5"  validation_file‚""  testing_file‚""  normalize_features‚1  limit_to_num_jets‚0  balance_particles‚0  balance_jets‚0 ! balance_classifications‚false  partial_events‚1  dataset_randomization‚0  train_validation_split‚0.95  limit_val_batches‚1.0  num_dataloader_workers‚0  mask_sequence_vectors‚1  combine_pair_loss‚"min"  optimizer‚"AdamW"  focal_gamma‚0.0  combinatorial_scale‚0.0 # learning_rate_warmup_epochs‚1.0  learning_rate_cycles‚1  assignment_loss_scale‚1.0  detection_loss_scale‚1.0  kl_loss_scale‚0.0  regression_loss_scale‚0.0 ! classification_loss_scale‚0.0 ! use_mass_classification‚false  mass_classes‚[]  balance_losses‚true  l2_penalty‚0.0002  gradient_clip‚10.0 num_gpu‚1  verbose_output‚false  usable_gpus‚""  trial_time‚"" $ trial_output_dir‚"./test_output"07Ù•ê n20ypkh2jØ §êÔ€ÖÔÈ/opt/conda/lib/python3.12/site-packages/pytorch_lightning/utilities/_pytree.py:21: `isinstance(treespec, LeafSpec)` is deprecated, use `isinstance(treespec, TreeSpec) and treespec.is_leaf()` instead. /In M n20ypkh2j» §êÔàôé«/opt/conda/lib/python3.12/site-packages/pytorch_lightning/trainer/connectors/data_connector.py:434: The 'val_dataloader' does not have many workers which may be a bottleneck. Consider increasing the value of the `num_workers` argument` to `num_workers=3` in the `DataLoader` to improve performance. YŽ—ë n20ypkh2jÙ ªêÔˆ¾­£È/opt/conda/lib/python3.12/site-packages/pytorch_lightning/utilities/_pytree.py:21: `isinstance(treespec, LeafSpec)` is deprecated, use `isinstance(treespec, TreeSpec) and treespec.is_leaf()` instead. ú>ÙaP n20ypkh2j¾ ªêÔ˜Ù¾£­/opt/conda/lib/python3.12/site-packages/pytorch_lightning/trainer/connectors/data_connector.py:434: The 'train_dataloader' does not have many workers which may be a bottleneck. Consider increasing the value of the `num_workers` argument` to `num_workers=3` in the `DataLoader` to improve performance. È ¬‘‚(:‡ µêÔôþ‡þ gpu.0.gpu‚4.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.195816040039062* gpu.0.memoryAllocatedBytes‚ 14771093504 gpu.0.temp‚36.0 gpu.0.powerWatts‚116.306' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚29.0765 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0V‚T1‚(:÷ µêÔÏüê€ disk.sdb.out‚ 1.91259765625 disk.sda2.out‚ 1.91259765625# proc.memory.rssMB‚ 2892.76171875 proc.cpu.threads‚17 disk.md0p1.out‚ 1.85546875 disk.md0.out‚ 1.85546875% disk./.usageGB‚2710.2438926696777 disk.sdb.in‚0 disk.sda.in‚0 disk.sda2.in‚0) proc.memory.percent‚17.65601634979248 network.recv‚42088 disk.nvme0n1.in‚ 0.02734375 disk.sdb1.in‚0 disk.md0.in‚0 disk.sdb2.in‚0 disk.rbd0.in‚0 disk.rbd2.out‚0 network.sent‚34706 disk.md0p1.in‚0 disk.sda.out‚ 1.91259765625 disk.rbd0.out‚0! disk.nvme1n1.out‚ 143.94921875 disk.sdb1.out‚0 disk.sdb2.out‚ 1.91259765625 disk.sda1.out‚0) disk./.usagePercent‚75.77955356423661 disk.sda1.in‚0 disk.dm-0.out‚144.1875$ memory_percent‚91.27168655395508' proc.memory.availableMB‚ 1430.046875 cpu‚25.922945015850864 disk.nvme0n1.out‚ 0.23828125 disk.nvme1n1.in‚ 0.05078125 disk.rbd2.in‚0 disk.dm-0.in‚0.078125­h]Ç’ ‚(:ˆ ÄêÔà°¡ƒ gpu.0.gpu‚57.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.195816040039062* gpu.0.memoryAllocatedBytes‚ 14771093504 gpu.0.temp‚37.0 gpu.0.powerWatts‚132.122' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚33.0305 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÎPä´ó!‚(:é ÄêÔ–Ç… cpu‚25.65949156627054% disk./.usageGB‚2710.3053550720215 disk.dm-0.out‚ 302.15234375 disk.md0p1.in‚0 disk.sdb.out‚ 3.6005859375 disk.sda.out‚ 3.6005859375" proc.memory.availableMB‚1428.5 disk.sdb2.out‚ 3.6005859375 disk.sda1.in‚0 disk.sda.in‚0# memory_percent‚91.2811279296875 disk.md0p1.out‚3.484375 disk.sdb.in‚0 network.sent‚50618 disk.nvme1n1.in‚ 0.05078125 disk.rbd2.out‚0 disk.sda2.in‚0 proc.memory.rssMB‚ 2892.8125 network.recv‚59754 disk.rbd2.in‚0 disk.nvme0n1.in‚ 0.02734375 disk.rbd0.out‚0 disk.nvme1n1.out‚ 251.9140625 disk.sda1.out‚0) disk./.usagePercent‚75.78127207873014 disk.rbd0.in‚0 disk.md0.out‚3.484375 disk.nvme0n1.out‚ 50.23828125 disk.sda2.out‚ 3.6005859375 proc.cpu.threads‚17 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.md0.in‚0 disk.sdb2.in‚0 disk.dm-0.in‚0.078125* proc.memory.percent‚17.656326293945312Úµ‘‘"‚(:‡ ÓêÔ›‹å gpu.0.gpu‚13.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.195816040039062* gpu.0.memoryAllocatedBytes‚ 14771093504 gpu.0.temp‚38.0 gpu.0.powerWatts‚163.09' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚40.7725 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÎÃ-ÿ#‚(:õ ÓêÔ”Æö„ disk.md0p1.in‚0 disk.sda2.out‚ 5.2900390625 disk.sdb2.out‚ 5.2900390625! proc.memory.rssMB‚ 2893.328125 disk.dm-0.in‚0.078125 disk.md0p1.out‚ 5.11328125 disk.sda.out‚ 5.2900390625 disk.sda2.in‚0 disk.sdb1.out‚0 disk.md0.out‚ 5.11328125 disk.sdb2.in‚0 proc.cpu.threads‚17 disk.dm-0.out‚ 591.10546875 disk.nvme0n1.in‚ 0.02734375 disk.md0.in‚0$ memory_percent‚91.28637313842773 network.sent‚65714! disk.nvme0n1.out‚ 150.57421875 disk.sda.in‚0 disk.rbd0.out‚0 disk.nvme1n1.out‚ 440.53125 disk.rbd2.out‚0% disk./.usageGB‚2710.5551147460938 cpu‚25.519473837403115 disk.sda1.in‚0 disk.sda1.out‚0 disk.sdb.out‚ 5.2900390625 disk.sdb1.in‚0) disk./.usagePercent‚75.78825546374975' proc.memory.availableMB‚ 1427.640625* proc.memory.percent‚17.659473419189453 network.recv‚77772 disk.sdb.in‚0 disk.rbd0.in‚0 disk.nvme1n1.in‚ 0.05078125 disk.rbd2.in‚03c€Ï3$b/ lr-AdamW/pg1"trainer/global_step2:JP˜‰7&3%b/ lr-AdamW/pg2"trainer/global_step2:JP”Î “-&) _wandbruntime‚59  _runtime‚59±®À'» & lr-AdamW/pg1‚3.476205907350504e-05 & lr-AdamW/pg2‚3.476205907350504e-05 trainer/global_step‚49 % _timestamp‚1.7885104336014605e+09 _runtime‚ 59.071980209 _step‚0ÌÜ#Â(½ _step‚0 & lr-AdamW/pg1‚3.476205907350504e-05 & lr-AdamW/pg2‚3.476205907350504e-05 trainer/global_step‚49 % _timestamp‚1.7885104336014605e+09 _runtime‚ 59.071980209¸a´Pl)bh EDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed"trainer/global_step2:JPðÂñ—j*bf CDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current"trainer/global_step2:JP‹(xüx+bt QDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated"trainer/global_step2:JP×Ï1zs,bo LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak"trainer/global_step2:JP#D a-b] :DeviceStatsMonitor.on_train_batch_start/num_oom_rejections"trainer/global_step2:JP)Âën.bj GDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed"trainer/global_step2:JPÅ3Vm/bi FDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak"trainer/global_step2:JP4–W0bS 0DeviceStatsMonitor.on_train_batch_start/num_ooms"trainer/global_step2:JPö¨Ôj1bf CDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated"trainer/global_step2:JP‘ä©k2bg DDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated"trainer/global_step2:JPèér3bn KDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated"trainer/global_step2:JP8An4bj GDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak"trainer/global_step2:JPs_iúl5bh EDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated"trainer/global_step2:JPqŽO©i6be BDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak"trainer/global_step2:JP+öƒ´g7bc @DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed"trainer/global_step2:JPHàžËn8bj GDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak"trainer/global_step2:JPÂK—p9bl IDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current"trainer/global_step2:JP£rGòp:bl IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated"trainer/global_step2:JPG Ìfs;bo LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated"trainer/global_step2:JPhëæˆmbe BDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current"trainer/global_step2:JP²ãJp?bl IDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current"trainer/global_step2:JP¡n]µo@bk HDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current"trainer/global_step2:JPû.Gø^AbZ 7DeviceStatsMonitor.on_train_batch_start/num_device_free"trainer/global_step2:JP¬’‰)tBbp MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed"trainer/global_step2:JP-RŸË`Cb\ 9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries"trainer/global_step2:JPš¿ïiDbe BDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current"trainer/global_step2:JP’C4éhEbd ADeviceStatsMonitor.on_train_batch_start/active.small_pool.current"trainer/global_step2:JPa§.ÂnFbj GDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak"trainer/global_step2:JPÙÔõíeGba >DeviceStatsMonitor.on_train_batch_start/allocation.all.current"trainer/global_step2:JP’ºs½bHb^ ;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak"trainer/global_step2:JPÊê=kIbg DDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated"trainer/global_step2:JP-{²^JbZ 7DeviceStatsMonitor.on_train_batch_start/active.all.peak"trainer/global_step2:JP-ßòûlKbh EDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed"trainer/global_step2:JPÏÈboLbk HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed"trainer/global_step2:JPº3Ò›gMbc @DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed"trainer/global_step2:JP9NØœpNbl IDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current"trainer/global_step2:JP´íåmObi FDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak"trainer/global_step2:JPþ®¹nPbj GDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed"trainer/global_step2:JPñ0‡iQbe BDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current"trainer/global_step2:JP3–oRbk HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed"trainer/global_step2:JPíðqSbm JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated"trainer/global_step2:JPSisˆhTbd ADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak"trainer/global_step2:JPy†£dUb` =DeviceStatsMonitor.on_train_batch_start/segment.all.allocated"trainer/global_step2:JPÖFÏeVba >DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak"trainer/global_step2:JPùÐndWb` =DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak"trainer/global_step2:JPˆ/ÒkXbg DDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak"trainer/global_step2:JPÙ$-@qYbm JDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current"trainer/global_step2:JPp¾„nZbj GDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed"trainer/global_step2:JP õïwg[bc @DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak"trainer/global_step2:JPJ}·n\bj GDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak"trainer/global_step2:JPÜþùe]ba >DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak"trainer/global_step2:JPj:Ùh^bd ADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed"trainer/global_step2:JP·²an_bj GDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated"trainer/global_step2:JP«)ìk`bg DDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current"trainer/global_step2:JPyÓwçgabc @DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak"trainer/global_step2:JP Aì6rbbn KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated"trainer/global_step2:JPÞÔ÷bfcbb ?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak"trainer/global_step2:JP™°¢ndbj GDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current"trainer/global_step2:JPÞVößjebf CDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed"trainer/global_step2:JP¢k”iefba >DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak"trainer/global_step2:JP‡é¦Ôpgbl IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated"trainer/global_step2:JP_ ³Rlhbh EDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current"trainer/global_step2:JP…~ˆ†sibo LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak"trainer/global_step2:JP`¢_jb[ 8DeviceStatsMonitor.on_train_batch_start/segment.all.peak"trainer/global_step2:JP’Yƒõfkbb ?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak"trainer/global_step2:JP;ô¢>mlbi FDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak"trainer/global_step2:JP’&Æ8gmbc @DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current"trainer/global_step2:JPßÖÑ(hnbd ADeviceStatsMonitor.on_train_batch_start/oversize_segments.current"trainer/global_step2:JPÐçiobe BDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current"trainer/global_step2:JPkéËQfpbb ?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak"trainer/global_step2:JP_‡pssqbo LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated"trainer/global_step2:JPenrbj GDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed"trainer/global_step2:JPðú`sb\ 9DeviceStatsMonitor.on_train_batch_start/segment.all.freed"trainer/global_step2:JP2ñQntbj GDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current"trainer/global_step2:JPöknqubm JDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current"trainer/global_step2:JPp/'gvbc @DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated"trainer/global_step2:JP5‚õ~rwbn KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated"trainer/global_step2:JPNÅDYgxbc @DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed"trainer/global_step2:JP@Òèkybg DDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated"trainer/global_step2:JPá«yjzbf CDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current"trainer/global_step2:JP‰"ƒk{bg DDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated"trainer/global_step2:JPWña|b] :DeviceStatsMonitor.on_train_batch_start/active.all.current"trainer/global_step2:JPÞeÃj}bf CDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed"trainer/global_step2:JPãl Ëf~bb ?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak"trainer/global_step2:JPÕOAobk HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed"trainer/global_step2:JP^·(c€b^ ;DeviceStatsMonitor.on_train_batch_start/segment.all.current"trainer/global_step2:JP‡`ülbg DDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak"trainer/global_step2:JPCjEÃw‚br ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current"trainer/global_step2:JP1BPuƒbp MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed"trainer/global_step2:JPqVd„b_ DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed"trainer/global_step2:JP¶;™!.¤) _wandbruntime‚59  _runtime‚59â»óö¥™O \JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚ 4545029147136 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1170 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13954823168 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1176279040 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897403872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13954823168 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1176279040 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 SADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚ 3975653025260 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚1247937 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510207109 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1180696576 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 898640896 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚1223347 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30186 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚ 4006936621568 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24590 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 242485760 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1170 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14218690560 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1180696576 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 916643840 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚563 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚ 4006936621568 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚2253493 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ODDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚911932 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914443232 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚563 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24592 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚1005554 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1141 XHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 55088233340 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1170 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ]MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 58517374464 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 4062215395328 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚ 4006918618624 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 916643840 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚ 4486284242944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚624853 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14218690560 $ _timestamp‚1.788510433602282e+09 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚624855 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 WEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚ 4063132039168 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 K@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚911319 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1174999685 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 YIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 56195417600 \LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 56195417600 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚613 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚611 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 251469824 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚ 4486260302848 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14218690560 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24590 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24592 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30431 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30431 UEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 55296776704 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚ 3920564791920 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1180696576 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1116 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚286466 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚563 XHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 55296776704ʬ£'¦O WEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚ 4006918618624 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 916643840 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚ 4486284242944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚624853 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14218690560 $ _timestamp‚1.788510433602282e+09 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚624855 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 WEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚ 4063132039168 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 K@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚911319 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1174999685 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 YIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 56195417600 \LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 56195417600 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚613 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚611 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 251469824 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚ 4486260302848 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14218690560 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24590 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24592 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30431 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30431 UEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 55296776704 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚ 3920564791920 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1180696576 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1116 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚286466 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚563 XHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 55296776704 \JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚ 4545029147136 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1170 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13954823168 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1176279040 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897403872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13954823168 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1176279040 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 SADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚ 3975653025260 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚1247937 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510207109 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1180696576 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 898640896 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚1223347 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30186 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚ 4006936621568 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24590 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 242485760 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1170 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14218690560 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1180696576 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 916643840 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚563 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚ 4006936621568 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚2253493 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ODDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚911932 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914443232 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚563 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24592 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚1005554 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1141 XHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 55088233340 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1170 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ]MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 58517374464 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 4062215395328pu†”§‚(:‰ âêÔƒÃሠgpu.0.gpu‚56.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.195816040039062* gpu.0.memoryAllocatedBytes‚ 14771093504 gpu.0.temp‚37.0 gpu.0.powerWatts‚117.525' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚29.38125 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0|hvÔ쨂(:á âêԲ͸’ disk.md0p1.in‚0 disk.sdb.in‚0* proc.memory.percent‚17.659640312194824 disk.nvme1n1.in‚ 0.05078125! disk.nvme1n1.out‚ 500.72265625 disk.md0.out‚ 6.7109375# proc.memory.rssMB‚ 2893.35546875 disk.sda2.out‚6.9375 disk.sdb1.out‚0 disk.md0p1.out‚ 6.7109375 network.sent‚83514 disk.sda.out‚6.9375$ memory_percent‚91.31290912628174 cpu‚25.431566167488594 proc.cpu.threads‚17 network.recv‚107868 disk.nvme0n1.in‚ 0.02734375 disk.nvme0n1.out‚ 250.5703125 disk.rbd0.in‚0) proc.memory.availableMB‚ 1423.29296875 disk.rbd2.in‚0 disk.sdb2.in‚0 disk.sdb2.out‚6.9375 disk.sda.in‚0) disk./.usagePercent‚75.79178358083128$ disk./.usageGB‚2710.681297302246 disk.md0.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.sdb.out‚6.9375 disk.sdb1.in‚0 disk.dm-0.in‚0.078125 disk.dm-0.out‚ 751.29296875 disk.sda2.in‚0 disk.rbd0.out‚0 disk.rbd2.out‚0’É¿Øm©bh EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed"trainer/global_step2:JPQiú,fªba >DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current"trainer/global_step2:JP +)k«bf CDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated"trainer/global_step2:JP9!'n¬bi FDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current"trainer/global_step2:JPž]eg­bb ?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak"trainer/global_step2:JP¸+h®bc @DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current"trainer/global_step2:JP%7óq¯bl IDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated"trainer/global_step2:JPR®+g°bb ?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed"trainer/global_step2:JP¥èe±b` =DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed"trainer/global_step2:JP>]Ëb²b] :DeviceStatsMonitor.on_train_batch_end/active.all.allocated"trainer/global_step2:JP®ÄÛg³bb ?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current"trainer/global_step2:JP¡ü‚m´bh EDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current"trainer/global_step2:JPþt¯rµbm JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated"trainer/global_step2:JP®ðšk¶bf CDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak"trainer/global_step2:JPòûUm·bh EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak"trainer/global_step2:JP? ~åk¸bf CDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated"trainer/global_step2:JPõÍ`g¹bb ?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed"trainer/global_step2:JP‰ÿø°bºb] :DeviceStatsMonitor.on_train_batch_end/allocation.all.freed"trainer/global_step2:JP>X|Ôq»bl IDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated"trainer/global_step2:JPÀ£Sr¼bm JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated"trainer/global_step2:JP æà«m½bh EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed"trainer/global_step2:JPLä!f¾ba >DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed"trainer/global_step2:JPýÔÐh¿bc @DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak"trainer/global_step2:JPP®×émÀbh EDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed"trainer/global_step2:JP¤.pÁbk HDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current"trainer/global_step2:JPSž»‚mÂbh EDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak"trainer/global_step2:JPDãÕqÃbl IDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated"trainer/global_step2:JPËd5ÛbÄb] :DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams"trainer/global_step2:JPVÈ„nÅbi FDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed"trainer/global_step2:JP÷ÙÇjÆbe BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated"trainer/global_step2:JPÇɉgÇbb ?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current"trainer/global_step2:JPä/YhÈbc @DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated"trainer/global_step2:JPj6:eÉb` =DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak"trainer/global_step2:JPî·ÓsÊbn KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed"trainer/global_step2:JPÒQÔiËbd ADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated"trainer/global_step2:JPᆻÔiÌbd ADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated"trainer/global_step2:JP´¼cdÍb_ DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed"trainer/global_step2:JP4û,Þeëb` =DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak"trainer/global_step2:JPÙ¤2liìbd ADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated"trainer/global_step2:JPí›3Dfíba >DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated"trainer/global_step2:JPá\îbW 4DeviceStatsMonitor.on_train_batch_end/max_split_size"trainer/global_step2:JPÚÿPjïbe BDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current"trainer/global_step2:JP‡ó?¨^ðbY 6DeviceStatsMonitor.on_train_batch_end/segment.all.peak"trainer/global_step2:JPÓ€‘kñbf CDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current"trainer/global_step2:JPsv¼hòbc @DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current"trainer/global_step2:JPUÑû¼qóbl IDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated"trainer/global_step2:JP⺅ødôb_ hHq]÷bX 5DeviceStatsMonitor.on_train_batch_end/num_device_free"trainer/global_step2:JPÒÒØÞhøbc @DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed"trainer/global_step2:JPŸ¶aùb\ 9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak"trainer/global_step2:JP8LL½dúb_ DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak"trainer/global_step2:JP‰]fþba >DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed"trainer/global_step2:JP„õ$÷lÿbg DDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak"trainer/global_step2:JP|w€br ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated"trainer/global_step2:JPÈ3ÙVbQ .DeviceStatsMonitor.on_train_batch_end/num_ooms"trainer/global_step2:JPMpÚ h‚bc @DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak"trainer/global_step2:JPÊçïDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak"trainer/global_step2:JPh“r½i‰bd ADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current"trainer/global_step2:JPLÕ¿jŠbe BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated"trainer/global_step2:JP¼Eh‹bc @DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current"trainer/global_step2:JP«õ%eŒb` =DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed"trainer/global_step2:JP[Æ‹ibd ADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed"trainer/global_step2:JPÑð-­mŽbh EDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed"trainer/global_step2:JPýE‹ibd ADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current"trainer/global_step2:JP ï¤pbk HDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current"trainer/global_step2:JP´gn‘bi FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed"trainer/global_step2:JP(#3Oj’be BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated"trainer/global_step2:JP‘yy+l“bg DDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak"trainer/global_step2:JPjïXd”b_ DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed"trainer/global_step2:JPêJ o™bj GDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated"trainer/global_step2:JP¥³Êkšbf CDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed"trainer/global_step2:JP„Ñpm›bh EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak"trainer/global_step2:JP ¾µ)lœbg DDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated"trainer/global_step2:JPØD_Vnbi FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed"trainer/global_step2:JP—|HGjžbe BDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak"trainer/global_step2:JPHO¦ì_ŸbZ 7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries"trainer/global_step2:JP½›õfe b` =DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed"trainer/global_step2:JPåã…¡m¡bh EDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak"trainer/global_step2:JPÍSò+a¢b\ 9DeviceStatsMonitor.on_train_batch_end/segment.all.current"trainer/global_step2:JP;êe£b` =DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak"trainer/global_step2:JP3EV.¤) _wandbruntime‚60  _runtime‚60«~‚È ¥¤M B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24618 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚2273390 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30433 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13954824192 [KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 59667993088 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 242485760 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14218690560 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚1025454 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚2298008 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1170 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24616 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚625 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚292716 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13954824192 I>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚929322 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1118 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚ 4575603463680 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚1272554 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚623 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 231710720 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 N>DeviceStatsMonitor.on_t˜õ?†ârain_batch_end/requested_bytes.all.peak‚ 13510207117 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 916657152 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚563 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14218690560 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚1025452 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚1025452 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚637229 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914443440 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 56409580032 MBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚929947 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚2 WGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 57308234240 SCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 56409580032 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚ 3998544886048 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1176280064 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1174999693 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1170 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 _runtime‚ 60.173895154 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 916657152 UCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚ 4143875151360 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 255650816 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 Q?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚ 4142958494208 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚1247938 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚2298008 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚ 4086566917120 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 394340352 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1176280064 UCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚ 4055656224146 Q?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚ 4054741780706 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚2273390 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚637231 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚ 3998561925408 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30188 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚292093 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1180696576 trainer/global_step‚49 % _timestamp‚1.7885104347028806e+09 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 56196894658 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14218690560 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24616 R@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚ 4143875151360 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1143 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚ 4575579523584 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚1272554 NDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚2 WGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 57308234240 SCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 56409580032 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚ 3998544886048 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1176280064 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1174999693 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1170 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 _runtime‚ 60.173895154 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 916657152 UCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚ 4143875151360 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 255650816 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 Q?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚ 4142958494208 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚1247938 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚2298008 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚ 4086566917120 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 394340352 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1176280064 UCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚ 4055656224146 Q?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚ 4054741780706 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚2273390 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚637231 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚ 3998561925408 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30188 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚292093 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1180696576 trainer/global_step‚49 % _timestamp‚1.7885104347028806e+09 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 56196894658 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14218690560 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24616 R@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚ 4143875151360 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1143 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚ 4575579523584 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚1272554 NDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14218690560 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚1025454 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚2298008 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1170 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24616 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚625 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚292716 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13954824192 I>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚929322 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1118 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚ 4575603463680 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚1272554 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚623 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 231710720 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510207117 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 916657152 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚563 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14218690560 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚1025452< êC§b> loss/SRqqt3/assignment_loss"trainer/global_step2:JPsòÎBB¨b= loss/SRqqt3/detection_loss"trainer/global_step2:JP“ÈyŠ@©b; loss/FBt2/detection_loss"trainer/global_step2:JPdÜîaAªb< loss/FRt2/assignment_loss"trainer/global_step2:JPo‚pC«b> loss/SRqqt4/assignment_loss"trainer/global_step2:JP loss/SRqqt2/assignment_loss"trainer/global_step2:JP^Ñ;ìB®b= loss/SRqqt4/detection_loss"trainer/global_step2:JP¡¾ÄÂA¯b< loss/FBt1/assignment_loss"trainer/global_step2:JPä6Îf@°b; loss/FRt1/detection_loss"trainer/global_step2:JP_¼ó@±b; loss/FRt3/detection_loss"trainer/global_step2:JPÀz@²b; loss/FRt4/detection_loss"trainer/global_step2:JP­5(B³b= loss/SRqqt2/detection_loss"trainer/global_step2:JP.š†ê@´b; loss/FBt1/detection_loss"trainer/global_step2:JPniže@µb; loss/FBt4/detection_loss"trainer/global_step2:JP’Ák[A¶b< loss/FRt1/assignment_loss"trainer/global_step2:JP¤ÙMA·b< loss/FRt3/assignment_loss"trainer/global_step2:JP9©—äB¸b= loss/SRqqt1/detection_loss"trainer/global_step2:JP‡Y@@¹b; loss/FBt3/detection_loss"trainer/global_step2:JP¸ g-ºb( epoch"trainer/global_step2:JPï2ª¢C»b> loss/SRqqt1/assignment_loss"trainer/global_step2:JPá–ÅA¼b< loss/FBt3/assignment_loss"trainer/global_step2:JP…;’ÎA½b< loss/FBt2/assignment_loss"trainer/global_step2:JPßø*A¾b< loss/FBt4/assignment_loss"trainer/global_step2:JPæ6¬@¿b; loss/FRt2/detection_loss"trainer/global_step2:JP(ræ¼7Àb2 loss/total_loss"trainer/global_step2:JP(N³‚.Á) _wandbruntime‚60  _runtime‚60Ç|FÂÀ 0loss/SRqqt3/detection_loss‚2.396898256950691 .loss/FBt2/detection_loss‚5.662248675090571 _step‚3 0loss/FRt2/assignment_loss‚7.3501639115182975 1loss/SRqqt4/assignment_loss‚5.413056539934735 /loss/FRt4/assignment_loss‚7.360199781564566 2loss/SRqqt2/assignment_loss‚5.4480621587662466 0loss/SRqqt4/detection_loss‚4.142674654029136 _runtime‚ 60.177975483 0loss/FBt1/assignment_loss‚2.6030089259147644 .loss/FRt1/detection_loss‚6.360847195020761 .loss/FRt3/detection_loss‚2.385650039901809 .loss/FRt4/detection_loss‚4.283956840252265 0loss/SRqqt2/detection_loss‚4.014492526295639 /loss/FBt1/detection_loss‚42.671977147459984 .loss/FBt4/detection_loss‚6.943050449490547 /loss/FRt1/assignment_loss‚7.236694892247518 /loss/FRt3/assignment_loss‚7.131526220412481 0loss/SRqqt1/detection_loss‚2.927986515419824 .loss/FBt3/detection_loss‚3.205816466745961 epoch‚0 1loss/SRqqt1/assignment_loss‚5.503782766205924 0loss/FBt3/assignment_loss‚2.6455781148827593 % _timestamp‚1.7885104347065954e+09 0loss/FBt2/assignment_loss‚2.6680180152257282 /loss/FBt4/assignment_loss‚2.659129238128662 .loss/FRt2/detection_loss‚4.459756841392894 &loss/total_loss‚150.76844523049147 /loss/SRqqt3/assignment_loss‚5.2938690576397 trainer/global_step‚49cÔë‚JÃÄ .loss/FBt3/detection_loss‚3.205816466745961 epoch‚0 1loss/SRqqt1/assignment_loss‚5.503782766205924 0loss/FBt3/assignment_loss‚2.6455781148827593 % _timestamp‚1.7885104347065954e+09 0loss/FBt2/assignment_loss‚2.6680180152257282 /loss/FBt4/assignment_loss‚2.659129238128662 .loss/FRt2/detection_loss‚4.459756841392894 &loss/total_loss‚150.76844523049147 /loss/SRqqt3/assignment_loss‚5.2938690576397 trainer/global_step‚49 0loss/SRqqt3/detection_loss‚2.396898256950691 .loss/FBt2/detection_loss‚5.662248675090571 _step‚3 0loss/FRt2/assignment_loss‚7.3501639115182975 1loss/SRqqt4/assignment_loss‚5.413056539934735 /loss/FRt4/assignment_loss‚7.360199781564566 2loss/SRqqt2/assignment_loss‚5.4480621587662466 0loss/SRqqt4/detection_loss‚4.142674654029136 _runtime‚ 60.177975483 0loss/FBt1/assignment_loss‚2.6030089259147644 .loss/FRt1/detection_loss‚6.360847195020761 .loss/FRt3/detection_loss‚2.385650039901809 .loss/FRt4/detection_loss‚4.283956840252265 0loss/SRqqt2/detection_loss‚4.014492526295639 /loss/FBt1/detection_loss‚42.671977147459984 .loss/FBt4/detection_loss‚6.943050449490547 /loss/FRt1/assignment_loss‚7.236694892247518 /loss/FRt3/assignment_loss‚7.131526220412481 0loss/SRqqt1/detection_loss‚2.927986515419824U¡”ñ”Ä‚(:‰ ñêÔžÒžþ gpu.0.gpu‚54.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.195816040039062* gpu.0.memoryAllocatedBytes‚ 14771093504 gpu.0.temp‚38.0 gpu.0.powerWatts‚170.219' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.55475 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0û€ˆûÅ‚(:ú ñêÔˆ…ùþ$ disk./.usageGB‚2710.150402069092 disk.sdb.in‚0 disk.sdb.out‚ 8.9775390625 disk.sda.out‚ 8.9775390625 disk.sda1.in‚0 disk.dm-0.out‚ 935.69921875 disk.md0p1.in‚0 disk.rbd0.out‚0 disk.rbd2.in‚0 disk.rbd2.out‚0# proc.memory.rssMB‚ 2893.35546875 network.sent‚117381 disk.nvme1n1.in‚ 0.05078125 disk.sdb1.out‚0* proc.memory.percent‚17.659640312194824) disk./.usagePercent‚75.77693952791536 disk.sda.in‚0 disk.rbd0.in‚0' proc.memory.availableMB‚ 1424.171875 disk.nvme0n1.out‚ 350.8828125 disk.sda1.out‚0 disk.dm-0.in‚0.078125$ memory_percent‚91.30754470825195 disk.md0.out‚ 8.68359375 disk.sdb2.in‚0 disk.sdb2.out‚ 8.9775390625 proc.cpu.threads‚17 network.recv‚115980 disk.sda2.out‚ 8.9775390625 disk.md0p1.out‚ 8.68359375 disk.nvme0n1.in‚ 0.02734375 disk.sda2.in‚0! disk.nvme1n1.out‚ 584.81640625 disk.sdb1.in‚0 disk.md0.in‚0 cpu‚25.372379077429432u"ê’Æ‚(:‡ €‚êÔÔ¯ÿ gpu.0.gpu‚56.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.195816040039062* gpu.0.memoryAllocatedBytes‚ 14771093504 gpu.0.temp‚38.0 gpu.0.powerWatts‚171.168' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.792 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0\órœÇ‚(:ö €‚êÔÔ”éÿ disk.sda.in‚0 disk.sdb2.in‚0 disk.sda1.out‚0 disk.sdb1.in‚0 disk.md0.in‚0( proc.memory.availableMB‚ 1423.9765625 proc.cpu.threads‚17 network.recv‚122015 disk.dm-0.in‚0.078125 disk.md0.out‚ 10.16015625( disk./.usagePercent‚75.7769313150544 disk.nvme0n1.in‚ 0.02734375 disk.sdb.out‚ 10.51171875 disk.sda2.in‚0 disk.sdb1.out‚0 disk.rbd2.in‚0 disk.sdb2.out‚ 10.51171875 disk.md0p1.in‚0 disk.md0p1.out‚ 10.16015625 disk.sda2.out‚ 10.51171875 disk.rbd0.in‚0 disk.nvme1n1.in‚ 0.05078125 cpu‚25.327906033064757 disk.nvme0n1.out‚450.875* proc.memory.percent‚17.659640312194824 network.sent‚159017% disk./.usageGB‚2710.1501083374023 disk.sdb.in‚0 disk.rbd0.out‚0 disk.rbd2.out‚0$ memory_percent‚91.30873680114746 disk.sda.out‚ 10.51171875! disk.nvme1n1.out‚ 645.51171875# proc.memory.rssMB‚ 2893.35546875 disk.sda1.in‚0 disk.dm-0.out‚ 1096.38671875ͦô’È‚(:‡ ‚êÔßðý gpu.0.gpu‚42.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.195816040039062* gpu.0.memoryAllocatedBytes‚ 14771093504 gpu.0.temp‚37.0 gpu.0.powerWatts‚166.548' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚41.637 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0~ É‚(:‚ ‚êÔ—‡îþ disk.sda1.in‚0 disk.md0.in‚0) disk./.usagePercent‚75.77518432219877 disk.sdb.in‚0 disk.sdb2.in‚0 disk.sdb2.out‚ 12.0380859375) proc.memory.availableMB‚ 1423.88671875 proc.cpu.threads‚17 disk.rbd2.out‚0% disk./.usageGB‚2710.0876274108887 disk.dm-0.in‚0.078125 disk.sda.out‚ 12.0380859375 disk.sda2.out‚ 12.0380859375 disk.rbd0.in‚0 disk.md0p1.in‚0 disk.sda.in‚0 disk.sdb1.in‚0* proc.memory.percent‚17.659640312194824 disk.dm-0.out‚ 1197.609375 disk.md0.out‚ 11.63671875 disk.rbd2.in‚0# proc.memory.rssMB‚ 2893.35546875 network.sent‚163212 disk.nvme0n1.in‚ 0.02734375 disk.sdb.out‚ 12.0380859375 disk.sda2.in‚0 disk.rbd0.out‚0! disk.nvme1n1.out‚ 645.83984375# memory_percent‚91.3092851638794 cpu‚25.292492134638323 disk.nvme1n1.in‚ 0.05078125 disk.sdb1.out‚0 network.recv‚126753 disk.md0p1.out‚ 11.63671875! disk.nvme0n1.out‚ 551.76953125 disk.sda1.out‚0Fn0Ê+ _wandbruntime‚116  _runtime‚116Œzñ9Â˼ trainer/global_step‚99 % _timestamp‚1.7885104912633474e+09 _runtime‚ 116.733905626 _step‚4 & lr-AdamW/pg1‚7.023354792402038e-05 & lr-AdamW/pg2‚7.023354792402038e-05TgÙ¾ÆÌÀ _step‚4 & lr-AdamW/pg1‚7.023354792402038e-05 & lr-AdamW/pg2‚7.023354792402038e-05 trainer/global_step‚99 % _timestamp‚1.7885104912633474e+09 _runtime‚ 116.733905626DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30188 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚570618 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 _runtime‚ 116.735492503 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13954824192 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚ 7988451399168 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 110951356928 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1174999693 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1170 % _timestamp‚1.7885104912645624e+09 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚2456249 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚563 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 245487616 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1170 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14218690560 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14218690560 trainer/global_step‚99 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24590 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚611 \JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚ 9068698109952 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚ 7819569498320 TBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚ 8100301396992 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 116229494784 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1180696576 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 916643840 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚ 7988433396224 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1180696576 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1163 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚ 8952221339648 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚ 7931018832000 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1170 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 916643840 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 111432294320 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1176280064 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1176280064 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚2000454 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24592 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚2000456 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚2480839 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚4481295 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1170 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1243655 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚ 7819586537680 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 HDeviceStatsMonitor.on_train_batch_start/oversize_segments.peŽ„ «Tak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚563 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 HDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚ 7988451399168 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 223335424 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 111849997824 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚1814273 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897403872 WEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚ 8100301396992 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚1813660 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1138 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914443232 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510207117 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1180696576 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 110951356928 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚2000456 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚2480839 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚2456249 SADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚ 8099384753152 XFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚ 9068450834432 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 116452830208 P>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 8099384753152 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 898640896 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24590 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30188 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚ 8952245279744 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 110534890448 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _step‚5 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 SADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚ 7930104388768 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24592 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13954824192 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚2000454 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 898640896 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 394340352 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096—„"¢±'Ï«O WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 HDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚ 7988451399168 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 223335424 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 111849997824 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚1814273 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897403872 WEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚ 8100301396992 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚1813660 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1138 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914443232 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510207117 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1180696576 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 110951356928 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚2000456 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚2480839 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚2456249 SADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚ 8099384753152 XFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚ 9068450834432 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 116452830208 P>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 8099384753152 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 898640896 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24590 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30188 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚ 8952245279744 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 110534890448 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _step‚5 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 SADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚ 7930104388768 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24592 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13954824192 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚2000454 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 898640896 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 394340352 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚563 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30188 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚570618 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 _runtime‚ 116.735492503 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13954824192 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚ 7988451399168 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 110951356928 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1174999693 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1170 % _timestamp‚1.7885104912645624e+09 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚2456249 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚563 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 245487616 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1170 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14218690560 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14218690560 trainer/global_step‚99 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24590 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚611 \JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚ 9068698109952 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚ 7819569498320 TBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚ 8100301396992 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 116229494784 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1180696576 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 916643840 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚ 7988433396224 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1180696576 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1163 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚ 8952221339648 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚ 7931018832000 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1170 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 916643840 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 111432294320 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1176280064 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1176280064 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚2000454 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24592 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚2000456 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚2480839 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚4481295 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1170 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1243655 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚ 7819586537680 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 HDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚563ý6±0Ð+ _wandbruntime‚117  _runtime‚117̯·&ѱM FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚1831729 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚563 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚2020352 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚ 9041540560384 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1170 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚576319 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 394340352 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚2020354 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24616 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1170 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚ 8068063691776 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚2505456 trainer/global_step‚99 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 916657152 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚ 9041564500480 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24618 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚4501192 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30433 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 112064160256 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚2480840 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚575700 ZHDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚ 9159174368768 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚ 7897566631808 NDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚4525810 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 898654208 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 Q?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚ 8009193144214 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1174999693 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1170 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1176280064 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚ 8068063691776 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 112962814464 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 112064160256 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚563 Q?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚ 8180127852032 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1138 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 117380254720 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 245487616 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚ 7897549592448 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 111643551766 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1180696576 _step‚6 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 253553664 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14218690560 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 112962814464 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30433 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1180696576 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚4525810 UCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚ 8010107587654 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24616 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚2020354 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510207117 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 112540955846 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1180696576 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1170 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚ 8068081694720 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13954824192 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30188 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1163 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚4501192 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14218690560 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚1831729 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚563 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚2020352 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚ 9041540560384 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1170 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚576319 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 394340352 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚2020354 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24616 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1170 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚ 8068063691776 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚2505456 trainer/global_step‚99 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 916657152 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚ 9041564500480 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24618 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚4501192 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30433 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 112064160256 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚2480840 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚575700 ZHDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚ 9159174368768 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚ 7897566631808 NDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚4525810 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 898654208 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 Q?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚ 8009193144214 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1174999693 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1170 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1176280064 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚ 8068063691776 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 112962814464 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 112064160256 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚563 Q?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚ 8180127852032 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1138 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 117380254720 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 245487616 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚ 7897549592448 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 111643551766 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1180696576 _step‚6 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 253553664 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14218690560 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 112962814464 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30433 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1180696576 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚4525810 UCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚ 8010107587654 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24616 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚2020354 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510207117 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 112540955846 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1180696576 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1170 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚ 8068081694720 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13954824192 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30188 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1163 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚4501192 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14218690560 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E-éÙ‚(:Þ ­‚êÔ¯ìÿ disk.nvme0n1.in‚ 0.02734375" proc.memory.rssMB‚ 2893.3671875 disk.nvme1n1.out‚741.4375( proc.memory.availableMB‚ 1423.5859375 disk.rbd0.out‚0 disk.sdb1.out‚0 disk.md0p1.out‚ 15.23828125! disk.nvme0n1.out‚ 766.01953125 disk.sdb.out‚15.75 disk.sdb1.in‚0 disk.md0.in‚0 disk.dm-0.in‚0.078125 disk.sda.in‚0 disk.sda2.in‚0 disk.rbd0.in‚0 disk.sda1.in‚0 disk.md0p1.in‚0 network.recv‚136264) disk./.usagePercent‚75.77777745305572 disk.rbd2.in‚0 disk.rbd2.out‚0 disk.sdb2.in‚0$ memory_percent‚91.31112098693848 cpu‚25.24213936125312 proc.cpu.threads‚17% disk./.usageGB‚2710.1803703308105 disk.sdb.in‚0 disk.sda2.out‚15.75* proc.memory.percent‚17.659711837768555 network.sent‚218529 disk.sda.out‚15.75 disk.sda1.out‚0 disk.nvme1n1.in‚ 0.05078125 disk.md0.out‚ 15.23828125 disk.sdb2.out‚15.75 disk.dm-0.out‚ 1507.45703125e=醒Ú‚(:‡ ¼‚êÔŸäôÿ gpu.0.gpu‚44.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.195816040039062* gpu.0.memoryAllocatedBytes‚ 14771093504 gpu.0.temp‚37.0 gpu.0.powerWatts‚130.564' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚32.641 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0å}z,÷Û‚(:ì ¼‚êÔšë€ disk.rbd2.in‚0 disk.sda1.out‚0 disk.nvme1n1.in‚ 0.05078125 disk.rbd2.out‚0 disk.dm-0.in‚0.078125 disk.nvme0n1.in‚ 0.02734375 disk.rbd0.in‚0 disk.sdb1.out‚0 network.sent‚225030 disk.sdb.in‚0 disk.sda.in‚0# memory_percent‚91.3118839263916 proc.cpu.threads‚17 disk.md0p1.out‚ 16.66015625 disk.nvme0n1.out‚ 802.1328125% disk./.usageGB‚2710.3055992126465 disk.sdb2.out‚17.21875 disk.dm-0.out‚ 1579.70703125 disk.sdb.out‚17.21875 disk.sda2.in‚0 disk.sdb1.in‚0( proc.memory.availableMB‚ 1423.4609375" proc.memory.rssMB‚ 2893.3671875 network.recv‚141455 disk.sda1.in‚0 disk.sda.out‚17.21875! disk.nvme1n1.out‚ 777.57421875* proc.memory.percent‚17.659711837768555( disk./.usagePercent‚75.7812789050042 disk.md0.out‚ 16.66015625 disk.md0p1.in‚0 disk.sda2.out‚17.21875 disk.md0.in‚0 disk.sdb2.in‚0 disk.rbd0.out‚0 cpu‚25.223983046657278$ +š“Ü‚(:ˆ Ë‚êÔ©®£þ gpu.0.gpu‚29.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.195816040039062* gpu.0.memoryAllocatedBytes‚ 14771093504 gpu.0.temp‚37.0 gpu.0.powerWatts‚163.422' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚40.8555 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ƒè° Ý‚(: Ë‚êÔËÒßÿ" proc.memory.rssMB‚ 2893.3671875 network.sent‚231462 network.recv‚146417 disk.sda2.out‚19.22998046875 disk.sda.out‚19.22998046875* proc.memory.percent‚17.659711837768555 disk.nvme1n1.in‚ 0.05078125 disk.nvme1n1.out‚849.8125 disk.sdb1.in‚0 disk.nvme0n1.in‚ 0.02734375" disk.nvme0n1.out‚ 1002.52734375 disk.sdb.in‚0) proc.memory.availableMB‚ 1423.53515625 disk.md0.in‚0 disk.sdb.out‚19.22998046875 disk.sda2.in‚0 disk.rbd0.out‚0 disk.sdb2.in‚0$ memory_percent‚91.31143093109131) disk./.usagePercent‚75.78475795824096% disk./.usageGB‚2710.4300270080566 disk.sdb1.out‚0 disk.rbd2.in‚0 disk.dm-0.out‚ 1852.33984375 disk.sdb2.out‚19.22998046875 disk.sda1.in‚0 disk.dm-0.in‚0.078125 disk.md0p1.in‚0 cpu‚25.207953966898888 disk.rbd0.in‚0 disk.md0.out‚ 18.609375 disk.md0p1.out‚ 18.609375 disk.sda.in‚0 proc.cpu.threads‚17 disk.rbd2.out‚0 disk.sda1.out‚0±w0Þ+ _wandbruntime‚174  _runtime‚174Õm²GÄß¾ ' lr-AdamW/pg1‚0.00010570503677453573 ' lr-AdamW/pg2‚0.00010570503677453573 trainer/global_step‚149 % _timestamp‚1.7885105495180645e+09 _runtime‚ 174.98918153 _step‚80‚LÐÈà ' lr-AdamW/pg2‚0.00010570503677453573 trainer/global_step‚149 % _timestamp‚1.7885105495180645e+09 _runtime‚ 174.98918153 _step‚8 ' lr-AdamW/pg1‚0.00010570503677453573ç+-S0á+ _wandbruntime‚174  _runtime‚174 95V½'â·O KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24590 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚3689151 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚11969948173824 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚12137470754816 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚11718591244080 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897403872 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14218690560 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚3713741 HDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1163 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30433 _runtime‚ 174.99136517 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24592 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1862455 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 394340352 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 166605937152 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1862453 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 253566976 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1180696576 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚2995356 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚12137470754816 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚12136554110976 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 898640896 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚3713741 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 245487616 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚11718574204720 % _timestamp‚1.7885105495202777e+09 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚853630 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30188 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 898640896 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _step‚9 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚11969966176768 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 166878951428 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14218690560 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 166605937152 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1176280064 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30433 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚13592480178688 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚13418182376448 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚13418206316544 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607†ßÁ'ã»O A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1170 HDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30188 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 898640896 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _step‚9 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚11969966176768 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 166878951428 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14218690560 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 166605937152 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1176280064 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30433 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚13592480178688 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚13418182376448 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚13418206316544 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24590 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚3689151 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚11969948173824 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚12137470754816 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚11718591244080 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897403872 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14218690560 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚3713741 HDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1163 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30433 _runtime‚ 174.99136517 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24592 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1862455 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 394340352 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 166605937152 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1862453 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 253566976 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1180696576 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚2995356 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚12137470754816 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚12136554110976 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 898640896 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚3713741 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 245487616 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚11718574204720 % _timestamp‚1.7885105495202777e+09 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚853630 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚6753612 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚589 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚563 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1180696576 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚6728994 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚859346 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 175207427584 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚11796554298848 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30433 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚3738358 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914443440 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚563 _step‚10 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24616 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 916657152 _runtime‚ 176.085477463 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897404080 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚12049596472320 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚13682709024768 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1174999693 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 168617394688 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚3713742 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚859935 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚3713742 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 898654208 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚3738358 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚2734175 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1138 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚13507525537280 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 245487616 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1176280064 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚2734766 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚6753612 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 394340352 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1170 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13954824192 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚12049578469376 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚6728994 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1180696576 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚12218213867008 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 167718740480 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30433 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30188 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚13507501597184 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚11796571338208 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚3015252 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24616 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1176280064 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1163 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1874829 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1170 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚563 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 168617394688 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚13682956286976 ODeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510207117 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14218690560 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚591 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1170 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1170 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚3015254 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096è²D"Í&æÇM VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897404080 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚12049596472320 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚13682709024768 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1174999693 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 168617394688 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚3713742 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚859935 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚3713742 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 898654208 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚3738358 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚2734175 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1138 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚13507525537280 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 245487616 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1176280064 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚2734766 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚6753612 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 394340352 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1170 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13954824192 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚12049578469376 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚6728994 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1180696576 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚12218213867008 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 167718740480 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30433 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30188 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚13507501597184 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚11796571338208 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚3015252 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24616 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1176280064 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1163 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1874829 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1170 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚563 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 168617394688 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚13682956286976 ODeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510207117 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14218690560 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚591 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1170 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1170 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚3015254 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚12049596472320 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 167718740480 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚6753612 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚589 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚563 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1180696576 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚6728994 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚859346 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 175207427584 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚11796554298848 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30433 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚3738358 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914443440 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚563 _step‚10 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24616 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 916657152 _runtime‚ 176.085477463 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 %qðe0ç+ _wandbruntime‚176  _runtime‚176¢—PèÊ 0loss/SRqqt1/detection_loss‚4.772627305043371 /loss/FBt2/detection_loss‚12.197090029716492 trainer/global_step‚149 /loss/FRt1/assignment_loss‚7.110835170745849 1loss/SRqqt4/assignment_loss‚3.787464348708882 0loss/FBt3/assignment_loss‚0.8161545049088696 /loss/FRt1/detection_loss‚3.4238583040237427 1loss/SRqqt2/detection_loss‚3.7906676570574445 %loss/total_loss‚98.18262410545208 _runtime‚ 176.086498158 /loss/FRt2/detection_loss‚4.0512923655055815 /loss/FBt3/detection_loss‚1.9210303772284851 .loss/FRt3/detection_loss‚2.778445816097351 0loss/SRqqt3/detection_loss‚4.077136434472688 1loss/SRqqt4/detection_loss‚1.6545398779768112 /loss/FBt4/detection_loss‚3.4347633162344042 2loss/SRqqt1/assignment_loss‚3.5063059329986572 0loss/FBt1/assignment_loss‚0.6843681582038037 /loss/FBt4/assignment_loss‚0.305015248098326 _step‚11 epoch‚0 $ _timestamp‚1.788510550616342e+09 /loss/FRt2/assignment_loss‚7.367503120785668 /loss/FRt3/assignment_loss‚7.100049413167513 /loss/FRt4/assignment_loss‚7.362787700834728 2loss/SRqqt2/assignment_loss‚3.2746149804857043 0loss/FBt2/assignment_loss‚0.3949988377114226 -loss/FRt4/detection_loss‚3.86869179900913 2loss/SRqqt3/assignment_loss‚3.2448273635492093 /loss/FBt1/detection_loss‚7.2575560428879475ÖöÇÀTéÎ 2loss/SRqqt1/assignment_loss‚3.5063059329986572 0loss/FBt1/assignment_loss‚0.6843681582038037 /loss/FBt4/assignment_loss‚0.305015248098326 _step‚11 epoch‚0 $ _timestamp‚1.788510550616342e+09 /loss/FRt2/assignment_loss‚7.367503120785668 /loss/FRt3/assignment_loss‚7.100049413167513 /loss/FRt4/assignment_loss‚7.362787700834728 2loss/SRqqt2/assignment_loss‚3.2746149804857043 0loss/FBt2/assignment_loss‚0.3949988377114226 -loss/FRt4/detection_loss‚3.86869179900913 2loss/SRqqt3/assignment_loss‚3.2448273635492093 /loss/FBt1/detection_loss‚7.2575560428879475 0loss/SRqqt1/detection_loss‚4.772627305043371 /loss/FBt2/detection_loss‚12.197090029716492 trainer/global_step‚149 /loss/FRt1/assignment_loss‚7.110835170745849 1loss/SRqqt4/assignment_loss‚3.787464348708882 0loss/FBt3/assignment_loss‚0.8161545049088696 /loss/FRt1/detection_loss‚3.4238583040237427 1loss/SRqqt2/detection_loss‚3.7906676570574445 %loss/total_loss‚98.18262410545208 _runtime‚ 176.086498158 /loss/FRt2/detection_loss‚4.0512923655055815 /loss/FBt3/detection_loss‚1.9210303772284851 .loss/FRt3/detection_loss‚2.778445816097351 0loss/SRqqt3/detection_loss‚4.077136434472688 1loss/SRqqt4/detection_loss‚1.6545398779768112 /loss/FBt4/detection_loss‚3.4347633162344042 †uñ”ê‚(:‰ Ú‚êÔÛ§§ÿ gpu.0.gpu‚45.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.195816040039062* gpu.0.memoryAllocatedBytes‚ 14771093504 gpu.0.temp‚37.0 gpu.0.powerWatts‚159.079' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚39.76975 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¢w8äñë‚(:æ Ú‚êÔ¾ë§% disk./.usageGB‚2710.4296531677246 disk.nvme1n1.in‚ 0.05078125 disk.sdb.in‚0 disk.sdb2.in‚0 disk.sda1.out‚0 disk.sda.in‚0 disk.sda.out‚ 20.666015625 disk.sda2.out‚ 20.666015625 disk.md0.out‚20 disk.sdb2.out‚ 20.666015625 disk.sda1.in‚0 disk.dm-0.out‚ 1924.3828125 disk.md0p1.out‚20 disk.nvme0n1.in‚ 0.02734375 disk.rbd0.out‚0 cpu‚25.19480089905418( disk./.usagePercent‚75.7847475055088 disk.nvme1n1.out‚ 921.8515625 disk.md0.in‚0 disk.md0p1.in‚0 disk.nvme0n1.out‚ 1002.53125 proc.cpu.threads‚17 network.recv‚151493 disk.sdb1.out‚0 disk.dm-0.in‚0.078125 disk.sdb.out‚ 20.666015625' proc.memory.availableMB‚ 1423.796875 disk.sdb1.in‚0$ memory_percent‚91.30983352661133" proc.memory.rssMB‚ 2893.3671875* proc.memory.percent‚17.659711837768555 network.sent‚251033 disk.rbd2.out‚0 disk.rbd0.in‚0 disk.rbd2.in‚0 disk.sda2.in‚0@ìuŠ”ì‚(:‰ é‚êÔ¥§êÿ gpu.0.gpu‚45.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.195816040039062* gpu.0.memoryAllocatedBytes‚ 14771093504 gpu.0.temp‚37.0 gpu.0.powerWatts‚161.587' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚40.39675 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0²¶×í‚(:… é‚êÔóˆŸ% disk./.usageGB‚2710.6796340942383 disk.rbd0.out‚0 disk.rbd2.out‚0 disk.dm-0.in‚0.078125" disk.nvme0n1.out‚ 1152.79296875 disk.sdb1.in‚0 disk.sda1.in‚0 disk.sdb.out‚22.40771484375) proc.memory.availableMB‚ 1424.48828125# proc.memory.rssMB‚ 2893.37109375 cpu‚25.18269124665729 disk.sda2.out‚22.40771484375 disk.nvme1n1.in‚ 0.05078125 disk.dm-0.out‚ 2183.0234375 disk.sda.in‚0 disk.md0.out‚ 21.671875 disk.sdb2.out‚22.40771484375 disk.sda1.out‚0 disk.sdb.in‚0 disk.sda.out‚22.40771484375 disk.sda2.in‚0" disk.nvme1n1.out‚ 1030.23046875 disk.md0.in‚0* proc.memory.percent‚17.659735679626465 network.sent‚286917 disk.sdb1.out‚0 disk.sdb2.in‚0 disk.md0p1.out‚ 21.671875 disk.nvme0n1.in‚ 0.02734375 proc.cpu.threads‚17 network.recv‚156091 disk.rbd0.in‚0 disk.rbd2.in‚0 disk.md0p1.in‚0$ memory_percent‚91.30561351776123) disk./.usagePercent‚75.79173707683927 ¥Ê6’î‚(:‡ ø‚êÔþǬþ gpu.0.gpu‚45.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.195816040039062* gpu.0.memoryAllocatedBytes‚ 14771093504 gpu.0.temp‚37.0 gpu.0.powerWatts‚124.044' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚31.011 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0´¤ÿï‚(:„ ø‚êÔÍ«öÿ disk.dm-0.out‚ 2255.05859375 disk.sdb.out‚23.94873046875 disk.md0.out‚ 23.16015625 cpu‚25.172407818629527 disk.sda1.in‚0 disk.sda1.out‚0 disk.nvme0n1.in‚ 0.02734375 disk.sdb1.out‚0 disk.nvme0n1.out‚ 1152.796875 disk.sda.in‚0) disk./.usagePercent‚75.79173707683927 disk.md0p1.out‚ 23.16015625 disk.sdb.in‚0 disk.sda.out‚23.94873046875 disk.rbd0.in‚0 disk.rbd2.out‚0% disk./.usageGB‚2710.6796340942383 disk.sda2.in‚0 disk.sda2.out‚23.94873046875 disk.nvme1n1.in‚ 0.05078125$ memory_percent‚91.30833148956299) proc.memory.availableMB‚ 1424.04296875* proc.memory.percent‚17.659759521484375 disk.dm-0.in‚0.078125 disk.sdb2.in‚0 disk.sdb2.out‚23.94873046875 proc.memory.rssMB‚2893.375 proc.cpu.threads‚17 disk.md0.in‚0 disk.rbd2.in‚0 network.sent‚293366 network.recv‚161097 disk.md0p1.in‚0 disk.rbd0.out‚0" disk.nvme1n1.out‚ 1102.26171875 disk.sdb1.in‚0 b”ð‚(:‰ ‡ƒêÔùã½ÿ gpu.0.gpu‚31.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.195816040039062* gpu.0.memoryAllocatedBytes‚ 14771093504 gpu.0.temp‚38.0 gpu.0.powerWatts‚166.207' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚41.55175 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0÷Ô ¢ùñ‚(:î ‡ƒêÔ’¢¦€ disk.sda.in‚0 disk.rbd2.out‚0 disk.dm-0.out‚ 2470.43359375 disk.nvme0n1.in‚ 0.02734375& proc.memory.availableMB‚ 1424.40625 disk.sda2.out‚ 25.43359375 disk.sda1.in‚0 disk.sda1.out‚0 disk.sdb.in‚0$ memory_percent‚91.30611419677734 network.sent‚299761 disk.dm-0.in‚0.078125 proc.memory.rssMB‚2893.375* proc.memory.percent‚17.659759521484375) disk./.usagePercent‚75.79173761014192% disk./.usageGB‚2710.6796531677246 disk.sda.out‚ 25.43359375 disk.md0.out‚24.59375 disk.rbd2.in‚0 cpu‚25.16499539666265 network.recv‚165896 disk.sdb1.out‚0 disk.md0p1.out‚24.59375 disk.sdb.out‚ 25.43359375 proc.cpu.threads‚17 disk.rbd0.in‚0 disk.sdb1.in‚0 disk.sdb2.in‚0 disk.nvme0n1.out‚ 1381.859375 disk.sda2.in‚0 disk.rbd0.out‚0 disk.nvme1n1.in‚ 0.05078125 disk.sdb2.out‚ 25.43359375" disk.nvme1n1.out‚ 1102.44921875 disk.md0.in‚0 disk.md0p1.in‚0œ#åî0ò+ _wandbruntime‚232  _runtime‚232Ñm¬TÆóÀ _step‚12 ' lr-AdamW/pg1‚0.00014117652562505107 ' lr-AdamW/pg2‚0.00014117652562505107 trainer/global_step‚199 % _timestamp‚1.7885106071139195e+09 _runtime‚ 232.584431182†pÊôÄ trainer/global_step‚199 % _timestamp‚1.7885106071139195e+09 _runtime‚ 232.584431182 _step‚12 ' lr-AdamW/pg1‚0.00014117652562505107 ' lr-AdamW/pg2‚0.00014117652562505107 æû+0õ+ _wandbruntime‚232  _runtime‚232Ôb³FÞöºO O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14218690560 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1140179 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914443232 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 221428204664 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚3990256 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 231781009408 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 222325608536 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚16173723468800 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 898640896 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1139581 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1180696576 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚8936899 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24592 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7885106071151946e+09 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚17884167353344 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510207117 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚563 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 222260517376 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚600 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚16173723468800 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1147 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2481255 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚598 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1170 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 898640896 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚4922053 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 245487616 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2481253 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1170 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30188 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚3620834 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 916643840 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1176280064 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30188 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚15839007115784 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚16174640112640 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚15951462951424 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 247275520 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚15951480954368M„ÎÄ'÷¾O TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚16173723468800 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 898640896 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1139581 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1180696576 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚8936899 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24592 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7885106071151946e+09 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚17884167353344 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510207117 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚563 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 222260517376 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚600 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚16173723468800 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1147 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2481255 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚598 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1170 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 898640896 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚4922053 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 245487616 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2481253 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1170 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30188 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚3620834 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 916643840 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1176280064 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30188 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚15839007115784 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚16174640112640 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚15951462951424 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 247275520 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚15951480954368 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14218690560 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1140179 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914443232 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 221428204664 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚3990256 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 231781009408 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 222325608536 2ÒØa0ø+ _wandbruntime‚233  _runtime‚233mclÍË&ùÅM ODeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚3638712 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚4946644 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 223373320704 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚18206405741056 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚17973462633984 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14218690560 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 247262208 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 232943107072 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 _step‚14 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚4010154 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1172 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1145083 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1176280064 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1145669 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚15919010314670 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 _runtime‚ 233.721605811 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚16031093246976 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚8956796 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚17973486574080 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 223322112 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚15918095871230 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚8956796 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1176280064 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚16255383224832 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24616 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚586 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚8981414 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 396436480 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 222536865982 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 916657152 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚563 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13954824192 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30188 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 245487616 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚563 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚15695559005248 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14218690560 trainer/global_step‚199 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚16031111249920 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 224271974912 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚15695576044608 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 223434270062 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510207117 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 898654208 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚588 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1170 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚4010152 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚4971260 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚16031093246976 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1180696576 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚16031111249920 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚4971260 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1180696576 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚3639300 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24616 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 233166429184 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13954824192 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1174999693 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14218690560 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚4946644 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2493631 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1147 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1180696576 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0»@é’< úÉM JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚563 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚15695559005248 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14218690560 trainer/global_step‚199 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚16031111249920 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 224271974912 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚15695576044608 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 223434270062 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510207117 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 898654208 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚588 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1170 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚4010152 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚4971260 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚16031093246976 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1180696576 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚16031111249920 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚4971260 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1180696576 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚3639300 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24616 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 233166429184 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13954824192 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1174999693 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14218690560 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚4946644 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2493631 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1147 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1180696576 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ODeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚3638712 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚4946644 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 223373320704 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚18206405741056 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚17973462633984 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14218690560 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 247262208 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 232943107072 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 _step‚14 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚4010154 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1172 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1145083 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1176280064 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1145669 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚15919010314670 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 _runtime‚ 233.721605811 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚16031093246976 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚8956796 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚17973486574080 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 223322112 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚15918095871230 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚8956796 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1176280064 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚16255383224832 MCDeviceStak(©“tsMonitor.on_train_batch_end/allocation.small_pool.current‚24616 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚586 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚8981414 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 396436480 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 222536865982 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 916657152 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚563 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13954824192 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30188 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 245487616ÏSÚ0û+ _wandbruntime‚233  _runtime‚233qÒDvLüÆ 0loss/FBt1/assignment_loss‚0.5596304909060044 1loss/FBt4/assignment_loss‚0.15054105446043034 1loss/SRqqt2/detection_loss‚3.0191345528832505 1loss/SRqqt3/detection_loss‚1.5505669007450342 .loss/FBt4/detection_loss‚2.242400057373508 2loss/SRqqt2/assignment_loss‚3.4483480666364943 .loss/FRt1/detection_loss‚4.539904282851652 /loss/FRt2/detection_loss‚4.2641656414056435 -loss/FRt3/detection_loss‚2.40673137716949 /loss/SRqqt4/detection_loss‚2.48347250353381 .loss/FRt1/assignment_loss‚7.28107872876254 /loss/FRt4/assignment_loss‚7.398687587844001 1loss/SRqqt4/assignment_loss‚3.776983996232351 .loss/FRt4/detection_loss‚4.502324773619573 1loss/SRqqt1/detection_loss‚3.0543910787864164 _runtime‚ 233.722895397 _step‚15 /loss/FRt2/assignment_loss‚7.424360055189866 0loss/SRqqt3/assignment_loss‚3.44292799949646 epoch‚0 trainer/global_step‚199 %loss/total_loss‚99.67340673269015 $ _timestamp‚1.788510608252557e+09 2loss/SRqqt1/assignment_loss‚3.4169695550745183 /loss/FBt1/detection_loss‚11.349778220057487 .loss/FBt3/detection_loss‚4.716532254929811 /loss/FRt3/assignment_loss‚7.157187978426616 0loss/FBt2/assignment_loss‚0.5861671367456438 0loss/FBt3/assignment_loss‚0.6730157955549657 /loss/FBt2/detection_loss‚10.228106644004583ÏÏ€PýÊ .loss/FRt1/assignment_loss‚7.28107872876254 /loss/FRt4/assignment_loss‚7.398687587844001 1loss/SRqqt4/assignment_loss‚3.776983996232351 .loss/FRt4/detection_loss‚4.502324773619573 1loss/SRqqt1/detection_loss‚3.0543910787864164 _runtime‚ 233.722895397 _step‚15 /loss/FRt2/assignment_loss‚7.424360055189866 0loss/SRqqt3/assignment_loss‚3.44292799949646 epoch‚0 trainer/global_step‚199 %loss/total_loss‚99.67340673269015 $ _timestamp‚1.788510608252557e+09 2loss/SRqqt1/assignment_loss‚3.4169695550745183 /loss/FBt1/detection_loss‚11.349778220057487 .loss/FBt3/detection_loss‚4.716532254929811 /loss/FRt3/assignment_loss‚7.157187978426616 0loss/FBt2/assignment_loss‚0.5861671367456438 0loss/FBt3/assignment_loss‚0.6730157955549657 /loss/FBt2/detection_loss‚10.228106644004583 0loss/FBt1/assignment_loss‚0.5596304909060044 1loss/FBt4/assignment_loss‚0.15054105446043034 1loss/SRqqt2/detection_loss‚3.0191345528832505 1loss/SRqqt3/detection_loss‚1.5505669007450342 .loss/FBt4/detection_loss‚2.242400057373508 2loss/SRqqt2/assignment_loss‚3.4483480666364943 .loss/FRt1/detection_loss‚4.539904282851652 /loss/FRt2/detection_loss‚4.2641656414056435 -loss/FRt3/detection_loss‚2.40673137716949 /loss/SRqqt4/detection_loss‚2.48347250353381€íPE”þ‚(:‰ –ƒêÔŠ°Ðý gpu.0.gpu‚56.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.195816040039062* gpu.0.memoryAllocatedBytes‚ 14771093504 gpu.0.temp‚38.0 gpu.0.powerWatts‚172.395' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.09875 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ý.ùÿ‚(:î –ƒêÔìÉìþ disk.sdb.out‚ 27.3427734375$ memory_percent‚91.30899906158447 cpu‚25.157840436720697 disk.nvme0n1.out‚ 1402.90625 disk.sda2.in‚0 disk.sda1.in‚0 disk.sdb1.in‚0 disk.rbd2.in‚0 disk.sdb2.in‚0 disk.nvme1n1.out‚1318.25 disk.md0.in‚0 disk.dm-0.in‚0.078125 disk.rbd0.out‚0 proc.memory.rssMB‚2893.375 network.sent‚306448 network.recv‚171206 disk.nvme0n1.in‚ 0.02734375 disk.rbd0.in‚0 disk.sdb1.out‚0 disk.sdb2.out‚ 27.3427734375 disk.sdb.in‚0 disk.nvme1n1.in‚ 0.05078125* proc.memory.percent‚17.659759521484375 disk.rbd2.out‚0) proc.memory.availableMB‚ 1423.93359375 proc.cpu.threads‚17% disk./.usageGB‚2710.0116806030273 disk.dm-0.out‚ 2721.15625 disk.md0p1.in‚0 disk.md0p1.out‚26.4375 disk.sda.in‚0 disk.sda.out‚ 27.3427734375 disk.sda2.out‚ 27.3427734375 disk.sda1.out‚0) disk./.usagePercent‚75.77306081766476 disk.md0.out‚26.4375™úŒe”€‚(:‰ ¥ƒêÔáÚãý gpu.0.gpu‚39.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.195816040039062* gpu.0.memoryAllocatedBytes‚ 14771093504 gpu.0.temp‚38.0 gpu.0.powerWatts‚169.333' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.33325 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚06‹¨‚(:ø ¥ƒêÔ²Îæþ proc.cpu.threads‚17! disk.nvme0n1.out‚ 1403.6484375 disk.sdb.in‚0" disk.nvme1n1.out‚ 1318.37890625 disk.sdb1.in‚0 disk.rbd2.out‚0 disk.sda1.in‚0) proc.memory.availableMB‚ 1424.18359375 disk.rbd0.in‚0 disk.nvme1n1.in‚ 0.05078125 disk.md0.out‚27.90625 disk.sdb2.out‚ 28.8642578125 disk.md0p1.out‚27.90625 disk.sdb.out‚ 28.8642578125 disk.sda.out‚ 28.8642578125 disk.rbd2.in‚0* proc.memory.percent‚17.659759521484375 disk.sdb1.out‚0 disk.sda1.out‚0 disk.dm-0.out‚ 2722.02734375$ memory_percent‚91.30747318267822 disk.md0.in‚0 disk.sdb2.in‚0 network.sent‚355185% disk./.usageGB‚2709.6000480651855 disk.nvme0n1.in‚ 0.02734375 disk.sda.in‚0 disk.sda2.in‚0 disk.rbd0.out‚0 cpu‚25.1502973442626 network.recv‚175904( disk./.usagePercent‚75.7615513996251 disk.sda2.out‚ 28.8642578125 disk.dm-0.in‚0.078125 disk.md0p1.in‚0 proc.memory.rssMB‚2893.375G  y”‚‚(:‰ ´ƒêÔ’¹Þÿ gpu.0.gpu‚14.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.195816040039062* gpu.0.memoryAllocatedBytes‚ 14771093504 gpu.0.temp‚38.0 gpu.0.powerWatts‚119.091' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚29.77275 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ªfþƒ‚(:û ´ƒêÔϵ™€" disk.nvme1n1.out‚ 1385.93359375 disk.sda.in‚0 disk.sda2.in‚0* proc.memory.percent‚17.659783363342285 disk.dm-0.out‚ 2873.1328125 proc.cpu.threads‚17 disk.sda.out‚ 30.41796875 disk.sdb2.out‚ 30.41796875) disk./.usagePercent‚75.75978435458944 disk.nvme1n1.in‚ 0.05078125 disk.sdb1.out‚0 disk.dm-0.in‚0.078125 disk.rbd0.out‚0 disk.rbd2.out‚0 disk.sda2.out‚ 30.41796875( proc.memory.availableMB‚ 1423.6640625# proc.memory.rssMB‚ 2893.37890625 disk.md0p1.in‚0 disk.md0p1.out‚ 29.40234375 network.sent‚363636$ memory_percent‚91.31064414978027 cpu‚25.14326200387877 disk.sdb.out‚ 30.41796875$ disk./.usageGB‚2709.536849975586 disk.sdb1.in‚0 disk.md0.in‚0 disk.nvme0n1.in‚ 0.02734375" disk.nvme0n1.out‚ 1487.19921875 disk.md0.out‚ 29.40234375 disk.sdb2.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.sdb.in‚0 disk.rbd0.in‚0 disk.rbd2.in‚0 network.recv‚180642Pµ»”„‚(:‰ ÃêÔ‰àØÿ gpu.0.gpu‚56.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.195816040039062* gpu.0.memoryAllocatedBytes‚ 14771093504 gpu.0.temp‚39.0 gpu.0.powerWatts‚174.903' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.72575 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ë³í‚ …‚(:€ ÃêÔ»´“€ disk.sdb2.in‚0 disk.md0.in‚0 cpu‚25.138763099133126 disk.dm-0.out‚ 2966.26953125 disk.sda.in‚0 disk.sda2.in‚0 disk.rbd0.in‚0# proc.memory.rssMB‚ 2893.37890625 disk.sdb2.out‚32.26708984375 disk.dm-0.in‚0.078125 disk.sda2.out‚32.26708984375 disk.rbd0.out‚0! disk.nvme1n1.out‚ 1386.1171875 disk.rbd2.in‚0 proc.cpu.threads‚17 disk.md0p1.out‚31.1875 disk.sda.out‚32.26708984375$ disk./.usageGB‚2709.540271759033 disk.sda1.out‚0 disk.sdb.in‚0 disk.nvme1n1.in‚ 0.05078125 disk.sdb1.out‚0$ memory_percent‚91.30759239196777 network.sent‚369799) disk./.usagePercent‚75.75988002908676 disk.md0p1.in‚0 disk.nvme0n1.in‚ 0.02734375 disk.rbd2.out‚0* proc.memory.percent‚17.659783363342285 network.recv‚185198 disk.sda1.in‚0 disk.sdb.out‚32.26708984375( proc.memory.availableMB‚ 1424.1640625" disk.nvme0n1.out‚ 1580.15234375 disk.sdb1.in‚0 disk.md0.out‚31.1875ê 0†+ _wandbruntime‚285  _runtime‚285¢ õsƇÀ trainer/global_step‚249 % _timestamp‚1.7885106602156184e+09 _runtime‚ 285.686000406 _step‚16 ' lr-AdamW/pg1‚0.00017664801447556642 ' lr-AdamW/pg2‚0.00017664801447556642eÑ:ÊˆÄ % _timestamp‚1.7885106602156184e+09 _runtime‚ 285.686000406 _step‚16 ' lr-AdamW/pg1‚0.00017664801447556642 ' lr-AdamW/pg2‚0.00017664801447556642 trainer/global_step‚249:ñ©50‰+ _wandbruntime‚285  _runtime‚285Áƒ¾éÄ'Š¾O ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 278813738496 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚19932977729024 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚11164701 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 276874861772 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚19932977729024 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 277915097600 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 898640896 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24592 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30188 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1176280064 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14218690560 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚563 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 245178368 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 289749614080 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 898640896 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510207117 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914443232 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1180696576 $ _timestamp‚1.788510660216746e+09 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚22640099242496 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚6154955 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚20210892826624 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚4985156 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚601 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1180696576 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1170 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚599 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚563 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24590 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30188 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚4525647 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1172 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 245487616 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14218690560 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30433 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚4985154 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 221238272 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚20211809470464 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24590 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚19794372922524 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897403872 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚19932995731968 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 916643840 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24592 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30188 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1176280064 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14218690560 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚563 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 245178368 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 289749614080 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 898640896 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510207117 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914443232 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1180696576 $ _timestamp‚1.788510660216746e+09 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚22640099242496 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚6154955 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚20210892826624 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚4985156 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚601 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1180696576 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1170 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚599 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚563 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24590 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30188 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚4525647 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1172™0Œ+ _wandbruntime‚286  _runtime‚286\:ó“ ÉM K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚11209216 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 278880927170 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚22439447610880 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 290918324736 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 217030656 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3112431 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1170 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24618 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚3112429 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 916657152 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14218690560 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚4543111 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚5005054 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 279926555136 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914443440 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚19872547234738 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13954824192 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚22730582966272 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897404080 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1170 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚563 % _timestamp‚1.7885106611671724e+09 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚6204162 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510207117 _step‚18 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 898654208 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 279926555136 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚22730341995520 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1180696576 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13954824192}JÓÓ&ŽÍM OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 916657152 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30433 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1147 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1170 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1174999693 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14218690560 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚11209216 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1431304 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚563 _runtime‚ 286.638390987 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚5005052 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚4543735 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13954824192 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚22730582966272 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897404080 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1170 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚563 % _timestamp‚1.7885106611671724e+09 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚6204162 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510207117 _step‚18 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 898654208 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 279926555136 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚22730341995520 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1180696576 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13954824192 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚11209216 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 278880927170 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚22439447610880 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 290918324736 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 217030656 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3112431 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1170 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24618 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚3112429 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 916657152 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14218690560 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚4543111 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚5005054 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 279926555136 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914443440 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚19872547234738 EDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30433 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1712173 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1170 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 332321518880 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14218690560 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚24248978828288 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3718855 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚26816065486848 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚23415588323920 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚5980056 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚24248062184448 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 916643840 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚27163737616384 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚26816089426944 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1180696576 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚7412447 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30188 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 396436480 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚5980056 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510207117 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚23914510509568 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24592 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚563ÀT?­É'ÃO ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3718855 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚26816065486848 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚23415588323920 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚5980056 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚24248062184448 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 916643840 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚27163737616384 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚26816089426944 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1180696576 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚7412447 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30188 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 396436480 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚5980056 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510207117 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚23914510509568 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24592 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚563 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 916643840 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚5431656 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 334468318720 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚5431026 trainer/global_step‚299 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚24248062184448 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 333569677824 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚630 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 347672129536 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚5980054 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30433 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1712173 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1170 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 332321518880 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14218690560 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚24248978828288 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0Í)–×0ž+ _wandbruntime‚338  _runtime‚338¥´èÏ&ŸÉM KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1170 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚563 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚563 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚5449157 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚23493568418048 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚5999954 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚7412448 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1174 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1149 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚26905408647680 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚26905384707584 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚563 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1180696576 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚23827913041686 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1170 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 916657152 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚23994122802176 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚618 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚27254473242112 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 349064594432 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897404080 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚24329721940480 ODeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510207117 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14218690560 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1176280064 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 249359360 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 333430180198 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 % _timestamp‚1.7885107131441493e+09 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 916657152 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 898654208 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13954824192 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚620 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3731231 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 225419264 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚23826998598246 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14218690560 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24616 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24616 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30188 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30433 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚13437018 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚5999954 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 898654208 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1176280064 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚5449777 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914443440 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚23493585457408 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚24328805283328 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚7412448 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1718546 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1180696576 _step‚22 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚13412400 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚5999952 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13954824192 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚23994122802176 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 335581135360 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚3731229 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 348839175168 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚24329721940480 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1174999693 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _runtime‚ 338.615191793¢cW¬# ÍM SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 898654208 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1176280064 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚5449777 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914443440 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚23493585457408 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚24328805283328 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚7412448 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1718546 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1180696576 _step‚22 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚13412400 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚5999952 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13954824192 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚23994122802176 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 335581135360 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚3731229 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 348839175168 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚24329721940480 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1174999693 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _runtime‚ 338.615191793 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1170 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚563 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚563 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚5449157 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚23493568418048 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚5999954 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚7412448 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1174 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1149 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚26905408647680 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚26905384707584 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚563 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1180696576 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚23827913041686 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1170 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 916657152 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚23994122802176 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚618 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚27254473242112 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 349064594432 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897404080 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚24329721940480 ODeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510207117 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14218690560 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1176280064 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 249359360 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 333430180198 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 % _timestamp‚1.7885107131441493e+09 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 916657152 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 898654208 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13954824192 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚620 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3731231 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 225419264 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚23826998598246 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14218690560 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24616 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24616 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30188 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30433 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?Devó†ãK'iceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚13437018 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚5999954 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2"³ÌŒ0¡+ _wandbruntime‚338  _runtime‚338§ NÖO¢É /loss/FRt4/assignment_loss‚7.300863853720731 1loss/SRqqt1/assignment_loss‚2.016468580563863 /loss/FRt4/detection_loss‚3.5050947832506756 -loss/FBt1/detection_loss‚9.33404378592968 /loss/FBt3/detection_loss‚10.457996400920802 .loss/FRt1/assignment_loss‚7.24052153462949 2loss/SRqqt4/assignment_loss‚3.2762119240230985 0loss/FBt2/assignment_loss‚0.5283521316773461 1loss/FBt3/assignment_loss‚0.49737074463203873 0loss/SRqqt3/detection_loss‚1.873939985396444 &loss/total_loss‚107.68013576974133 epoch‚0 trainer/global_step‚299 _runtime‚ 338.61666292 /loss/FRt2/assignment_loss‚7.299921413637557 .loss/FRt2/detection_loss‚2.873340220102724 .loss/FRt3/detection_loss‚2.212682956829667 0loss/SRqqt1/detection_loss‚3.579025911622577 1loss/SRqqt4/detection_loss‚18.088379979133606 % _timestamp‚1.7885107131464837e+09 2loss/SRqqt2/assignment_loss‚3.2895177849407853 2loss/SRqqt3/assignment_loss‚3.0275684893131256 0loss/FBt1/assignment_loss‚0.6806711150165938 .loss/FBt2/detection_loss‚6.471918827813605 _step‚23 1loss/SRqqt2/detection_loss‚2.6651073845809905 /loss/FRt3/assignment_loss‚7.212911136448383 /loss/FBt4/assignment_loss‚0.276720958322015 /loss/FRt1/detection_loss‚3.0938391478165337 /loss/FBt4/detection_loss‚0.8776667194190062x$öS£Í 2loss/SRqqt2/assignment_loss‚3.2895177849407853 2loss/SRqqt3/assignment_loss‚3.0275684893131256 0loss/FBt1/assignment_loss‚0.6806711150165938 .loss/FBt2/detection_loss‚6.471918827813605 _step‚23 1loss/SRqqt2/detection_loss‚2.6651073845809905 /loss/FRt3/assignment_loss‚7.212911136448383 /loss/FBt4/assignment_loss‚0.276720958322015 /loss/FRt1/detection_loss‚3.0938391478165337 /loss/FBt4/detection_loss‚0.8776667194190062 /loss/FRt4/assignment_loss‚7.300863853720731 1loss/SRqqt1/assignment_loss‚2.016468580563863 /loss/FRt4/detection_loss‚3.5050947832506756 -loss/FBt1/detection_loss‚9.33404378592968 /loss/FBt3/detection_loss‚10.457996400920802 .loss/FRt1/assignment_loss‚7.24052153462949 2loss/SRqqt4/assignment_loss‚3.2762119240230985 0loss/FBt2/assignment_loss‚0.5283521316773461 1loss/FBt3/assignment_loss‚0.49737074463203873 0loss/SRqqt3/detection_loss‚1.873939985396444 &loss/total_loss‚107.68013576974133 epoch‚0 trainer/global_step‚299 _runtime‚ 338.61666292 /loss/FRt2/assignment_loss‚7.299921413637557 .loss/FRt2/detection_loss‚2.873340220102724 .loss/FRt3/detection_loss‚2.212682956829667 0loss/SRqqt1/detection_loss‚3.579025911622577 1loss/SRqqt4/detection_loss‚18.088379979133606 % _timestamp‚1.7885107131464837e+09£JÓŸ’¤‚(:‡ ÿƒêÔ¨íßÿ gpu.0.gpu‚55.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.195816040039062* gpu.0.memoryAllocatedBytes‚ 14771093504 gpu.0.temp‚39.0 gpu.0.powerWatts‚163.09' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚40.7725 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0fÍ%_¥‚(:ˆ ÿƒêÔÞžøÿ disk.sdb1.out‚0$ disk./.usageGB‚2710.249568939209 disk.rbd2.in‚0 disk.md0p1.in‚0 disk.rbd0.out‚0$ memory_percent‚91.30308628082275 disk.sda2.out‚38.78076171875 disk.md0p1.out‚ 37.48046875) disk./.usagePercent‚75.77971227510841 disk.rbd2.out‚0 disk.sdb2.in‚0 disk.nvme0n1.in‚ 0.02734375 disk.sdb.in‚0 disk.md0.in‚0 disk.md0.out‚ 37.48046875 disk.dm-0.out‚ 3332.12109375 disk.sda1.in‚0 disk.sdb.out‚38.78076171875 disk.sda2.in‚0 disk.rbd0.in‚0 disk.nvme1n1.in‚ 0.05078125* proc.memory.percent‚17.659783363342285 cpu‚25.120378129478855 disk.sda.in‚0) proc.memory.availableMB‚ 1424.90234375# proc.memory.rssMB‚ 2893.37890625 disk.dm-0.in‚0.078125" disk.nvme0n1.out‚ 1945.01171875 disk.sda.out‚38.78076171875 proc.cpu.threads‚17 disk.sdb1.in‚0 network.sent‚434506 network.recv‚203170 disk.sdb2.out‚38.78076171875 disk.sda1.out‚0 disk.nvme1n1.out‚ 1387.109375ïÏ™‘¦‚(:† Ž„êÔÅßÊÿ gpu.0.gpu‚19.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.195816040039062* gpu.0.memoryAllocatedBytes‚ 14771093504 gpu.0.temp‚37.0 gpu.0.powerWatts‚114.74' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.685 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ïû– §‚(:€ Ž„êÔìóÿ€% disk./.usageGB‚2710.3110313415527 disk.md0p1.in‚0 disk.nvme0n1.in‚ 0.02734375 disk.sda2.in‚0) disk./.usagePercent‚75.78143078960194 disk.sda.in‚0 disk.nvme1n1.in‚ 0.05078125 disk.rbd2.out‚0 disk.sda1.in‚0 disk.sda.out‚ 40.6826171875 disk.sda2.out‚ 40.6826171875 cpu‚25.117483622590115 disk.md0p1.out‚ 39.32421875 disk.nvme0n1.out‚ 2138.921875 disk.sdb.out‚ 40.6826171875 disk.rbd0.out‚0 disk.nvme1n1.out‚ 1387.34375 network.sent‚484257 disk.sdb2.in‚0 disk.sda1.out‚0 disk.sdb1.in‚0# proc.memory.rssMB‚ 2893.37890625 disk.sdb2.out‚ 40.6826171875 disk.dm-0.in‚0.078125 disk.dm-0.out‚ 3526.265625 disk.rbd0.in‚0 disk.sdb1.out‚0$ memory_percent‚91.30032062530518 disk.sdb.in‚0 disk.md0.in‚0 disk.md0.out‚ 39.32421875 network.recv‚209290 disk.rbd2.in‚0) proc.memory.availableMB‚ 1425.35546875* proc.memory.percent‚17.659783363342285 proc.cpu.threads‚17ƒîó!”¨‚(:‰ „êÔ„¾´‚ gpu.0.gpu‚55.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.195816040039062* gpu.0.memoryAllocatedBytes‚ 14771093504 gpu.0.temp‚38.0 gpu.0.powerWatts‚170.891' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.72275 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0c öñ©‚(:‰ „êÔŒû‹„ network.recv‚212991 disk.nvme1n1.in‚ 0.05078125 network.sent‚489971) disk./.usagePercent‚75.78491837568126 disk.nvme0n1.in‚ 0.02734375 disk.sdb.in‚0 disk.sda.in‚0 disk.sda2.out‚42.17626953125 disk.rbd0.out‚0 disk.sda1.in‚0$ disk./.usageGB‚2710.435764312744 disk.rbd2.in‚0 disk.md0p1.in‚0$ memory_percent‚91.30184650421143) proc.memory.availableMB‚ 1425.10546875# proc.memory.rssMB‚ 2893.37890625 proc.cpu.threads‚17 disk.sdb.out‚42.17626953125 disk.sdb1.in‚0 disk.rbd2.out‚0 disk.sdb2.out‚42.17626953125 disk.dm-0.out‚ 3746.4296875 disk.sda2.in‚0" disk.nvme1n1.out‚ 1387.48046875 disk.sdb1.out‚0 disk.sdb2.in‚0 cpu‚25.114552518927063 disk.md0.out‚ 40.76953125 disk.sda1.out‚0" disk.nvme0n1.out‚ 2358.94921875 disk.sda.out‚42.17626953125 disk.rbd0.in‚0 disk.md0.in‚0 disk.dm-0.in‚0.078125 disk.md0p1.out‚ 40.76953125* proc.memory.percent‚17.659783363342285,Ô©O’ª‚(:‡ ¬„êÔÚÿ gpu.0.gpu‚56.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.195816040039062* gpu.0.memoryAllocatedBytes‚ 14771093504 gpu.0.temp‚38.0 gpu.0.powerWatts‚175.512' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.878 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ØÝJ± «‚(:ÿ ¬„êÔÛ€€ disk.rbd0.in‚0 disk.sdb1.out‚0 disk.sda1.in‚0 disk.sdb.out‚ 43.572265625 disk.md0.out‚ 42.09765625 disk.sda.out‚ 43.572265625 disk.sda2.in‚0 disk.rbd0.out‚0 disk.nvme0n1.out‚ 2518.765625 disk.sdb.in‚0$ disk./.usageGB‚2710.686923980713 disk.rbd2.in‚0 disk.md0p1.out‚ 42.09765625$ memory_percent‚91.30187034606934( proc.memory.availableMB‚ 1425.1015625 network.recv‚217928 disk.nvme1n1.in‚ 0.05078125" disk.nvme1n1.out‚ 1387.69921875 disk.sdb1.in‚0 disk.sda1.out‚0 disk.dm-0.in‚0.078125 disk.md0p1.in‚0) disk./.usagePercent‚75.79194090511615 disk.sdb2.out‚ 43.572265625 disk.sda.in‚0* proc.memory.percent‚17.659783363342285 cpu‚25.111530208415346 disk.sda2.out‚ 43.572265625 disk.sdb2.in‚0 disk.nvme0n1.in‚ 0.02734375# proc.memory.rssMB‚ 2893.37890625 network.sent‚496093 disk.md0.in‚0 disk.rbd2.out‚0 disk.dm-0.out‚ 3906.46484375 proc.cpu.threads‚17(ÎÕö0¬+ _wandbruntime‚393  _runtime‚393%Ê(ÇÄ­¾ & lr-AdamW/pg1‚0.0002475909921765971 & lr-AdamW/pg2‚0.0002475909921765971 trainer/global_step‚349 % _timestamp‚1.7885107681306875e+09 _runtime‚ 393.601101545 _step‚24§dhȮ % _timestamp‚1.7885107681306875e+09 _runtime‚ 393.601101545 _step‚24 & lr-AdamW/pg1‚0.0002475909921765971 & lr-AdamW/pg2‚0.0002475909921765971 trainer/global_step‚349‡Ó^M0¯+ _wandbruntime‚393  _runtime‚393sU-TÅ'°¿O XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚27703275649540 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚8620759 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚6974956 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 916643840 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 396436480 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚563 _step‚25 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 898640896 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚31687978333184 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1174999693 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14218690560 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1170 trainer/global_step‚349 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13954824192 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚622 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚27896025287168 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914443232 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 389224258048 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚563 _runtime‚ 393.602477251 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 390122898944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1176280064 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14218690560 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 390122898944 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚15595713 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24592 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚31282026523648 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14218690560 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚6974954 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30188 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚15620305 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1999929 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚28286148186112 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚624 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚28285231542272 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 405710825472 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24590 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚6337584 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30433 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24590 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚27896025287168 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 389224258048 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 240984064 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚6974954 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚8645349 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1174 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24592 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚28285231542272 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚8620759 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚27314593030320 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚6336960 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚31687737349120 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1170 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1180696576 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4337653 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚6974956 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚28286148186112 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1170 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 217043968 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24592 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚31282026523648 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14218690560 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚6974954 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30188 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚15620305 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1999929 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚28286148186112 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚624 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚28285231542272 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 405710825472 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24590 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚6337584 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30433 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24590 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚27896025287168 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 389224258048 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 240984064 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚6974954 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚8645349 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1174 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24592 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚28285231542272 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚8620759 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚27314593030320 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚6336960 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚31687737349120 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1170 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1180696576 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4337653 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚6974956 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚28286148186112 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1170 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 217043968 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚563 _step‚25 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 898640896 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚31687978333184 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1174999693 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14218690560`çR0²+ _wandbruntime‚394  _runtime‚394p_QÿL ³ÉM MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚611 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1180696576 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510207117 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚27781449961754 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24616 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚31778464320000 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚6994854 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚6994854 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚6355116 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1180696576 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1170 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 _step‚26 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 916657152 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 % _timestamp‚1.7885107691280167e+09 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914443440 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚27392590163808 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14218690560 I=DeviceStatsMonitor.2öMƒon_train_batch_end/active.small_pool.freed‚8645350 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚31371369684480 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚563 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 406865021952 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1174999693 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚8669966 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13954824192 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚28366891298304 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 898654208 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚31778210766336 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 389774241386 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1170 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚27975637579776 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30433 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚6355729 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 391235715584 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚613 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 388876837306 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 229613568 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 246679040 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14218690560 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4350031 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚4350029 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚27392573124448 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 898654208 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 253553664 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 916657152 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚15664820 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚8669966 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚15640202 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚31371345744384 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24618 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1170 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1180696576 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚15640202 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30433 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 390337061376 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2005698 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24616 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚15664820 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚27782364405194 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚6994852 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 390337061376 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13954824192 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1176280064 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2005087 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 396436480 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚563 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚6994852 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897404080 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14218690560 trainer/global_step‚349Icú•Ó&´ÍM O8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4350031 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚4350029 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚27392573124448 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 898654208 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 253553664 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 916657152 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚15664820 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚8669966 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚15640202 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚31371345744384 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24618 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1170 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1180696576 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚15640202 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30433 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 390337061376 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2005698 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24616 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚15664820 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚27782364405194 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚6994852 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 390337061376 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13954824192 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1176280064 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2005087 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 396436480 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚563 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚6994852 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897404080 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14218690560 trainer/global_step‚349 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚611 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1180696576 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510207117 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚27781449961754 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24616 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚31778464320000 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚6994854 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚6994854 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚6355116 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1180696576 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1170 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 _step‚26 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 916657152 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 % _timestamp‚1.7885107691280167e+09 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914443440 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚27392590163808 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14218690560 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚8645350 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚31371369684480 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚563 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 406865021952 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1174999693 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚8669966 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13954824192 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚28366891298304 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 898654208 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚31778210766336 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 389774241386 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1170 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚27975637579776 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30433 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚6355729 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 391235715584 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚613 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 388876837306 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 229613568 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 246679040 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14218690560 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 EDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚32322400900096 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚619 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚7243227 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚9878251 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚31877540064768 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚617 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚36211900660224 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 464134338048 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚31656812569816 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 _runtime‚ 449.619052294 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1174999693 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 444878838272 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1170 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 916643840 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13954824192 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚31877522061824 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 898640896 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14218690560 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1180696576 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚17848107 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚31213614776080 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚31877522061824 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 444878838272 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4956455 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510207117 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14218690560 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24590 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚36212145838592 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 445777479168 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 221238272 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914443232 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897403872 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚563 ˜+\R+ _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚31877540064768 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24592 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 396436480 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1170 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24590 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 463913099776 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1170 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚32322400900096 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚7242608 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1174 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 245178368 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30188 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚32323317543936 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2286772 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 444112236968 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1176280064 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚7969856 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚7969854 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1149#4™È'ÃÂO TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14218690560 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1180696576 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚17848107 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚31213614776080 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚31877522061824 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 444878838272 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4956455 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510207117 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14218690560 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24590 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚36212145838592 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 445777479168 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 221238272 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914443232 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897403872 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚563 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚31877540064768 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24592 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 396436480 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1170 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24590 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 463913099776 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1170 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚32322400900096 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚7242608 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1174 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 245178368 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30188 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚32323317543936 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2286772 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 444112236968 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1176280064 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚7969856 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚7969854 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1149 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚31657727013048 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1180696576 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚563 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30433 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 445777479168 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚31213597736720 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 443214833096 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1176280064 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 trainer/global_step‚399 $ _timestamp‚1.788510824148268e+09 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚35748011500544 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 898640896 _step‚29 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚32322400900096 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚619 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚7243227 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚9878251 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚31877540064768 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚617 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚36211900660224 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 464134338048 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚31656812569816 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 _runtime‚ 449.619052294 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1174999693 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 444878838272 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1170 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 916643840 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13954824192 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚31877522061824 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 898640896 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0Tp9¾’Ä‚(:‡ è„êÔÊÒ§€ gpu.0.gpu‚34.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.195816040039062* gpu.0.memoryAllocatedBytes‚ 14771093504 gpu.0.temp‚38.0 gpu.0.powerWatts‚164.372' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚41.093 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ül¤Å‚(:… è„êÔ¨Ô disk.sda.out‚50.29736328125 disk.nvme1n1.in‚ 0.05078125 disk.sdb1.in‚0* proc.memory.percent‚17.659831047058105 network.recv‚236865$ disk./.usageGB‚2710.170795440674 disk.rbd2.out‚0 disk.sdb2.in‚0 disk.md0.out‚ 48.5703125 proc.cpu.threads‚17 disk.dm-0.out‚ 4546.734375 disk.nvme0n1.in‚ 0.02734375" disk.nvme1n1.out‚ 1466.03515625# memory_percent‚91.3015604019165) disk./.usagePercent‚75.77750973512009 disk.sda.in‚0 disk.sda2.in‚0 disk.rbd0.out‚0 disk.md0p1.out‚ 48.5703125 disk.sda2.out‚50.29736328125) proc.memory.availableMB‚ 1425.15234375# proc.memory.rssMB‚ 2893.38671875 disk.sdb2.out‚50.29736328125 disk.dm-0.in‚0.078125 disk.md0p1.in‚0 disk.sdb.in‚0 disk.md0.in‚0 disk.rbd0.in‚0 cpu‚25.102022186583202 disk.rbd2.in‚0" disk.nvme0n1.out‚ 3080.69921875 disk.sdb.out‚50.29736328125 disk.sdb1.out‚0 network.sent‚564437 disk.sda1.in‚0 disk.sda1.out‚0L¤60Æ+ _wandbruntime‚450  _runtime‚450W«Ï&ÇÉM ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚615 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 226290688 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚32403143998976 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1174999693 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1180696576 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚ 450.657050999 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚7989754 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚17892622 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2291876 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚36302644599808 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 246679040 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 916657152 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚31957152357376 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 898654208 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚7260705 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚563 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30188 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚35837330721280 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 446890295808 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚7989754 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚617 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚17892622 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚17868004 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚32404060656128 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 445991641600 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚32404060656128 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510207117 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚563 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1176280064 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13954824192 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897404080 _step‚30 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2292491 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 202350592 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚31291577830848 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14218690560 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚17868004 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 445991641600 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚36302418309120 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914443440 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1180696576 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚9902868 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24616 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1180696576 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 898654208Ž/<Ž;!ÈÍM ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚31957170360320 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 446890295808 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚ 450.657050999 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚7989754 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚17892622 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2291876 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚36302644599808 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 246679040 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 916657152 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚31957152357376 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 898654208 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚7260705 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚563 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30188 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚35837330721280 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 446890295808 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚7989754 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚617 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚17892622 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚17868004 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚32404060656128 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 445991641600 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚32404060656128 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510207117 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚563 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1176280064 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13954824192 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897404080 _step‚30 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2292491 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 202350592 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚31291577830848 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14218690560 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚17868004 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 445991641600 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚36302418309120 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914443440 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1180696576 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚9902868 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24616 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1180696576 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 898654208 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚615 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 226290688 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚32403143998976 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1174999693 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1180696576 E˜57152357376 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1176280064 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30433 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚7261322 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1170 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24618 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30433 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚31735901325262 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14218690560 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚7989752 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 465313878528 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚31736815768702 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 445220898494 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944Ï-0É+ _wandbruntime‚450  _runtime‚450ÎB%RÊÌ 1loss/SRqqt4/assignment_loss‚3.482253543792232 0loss/FBt1/assignment_loss‚0.4196942813068485 .loss/FRt1/detection_loss‚3.408438966192048 .loss/FRt4/detection_loss‚4.517606518098286 0loss/SRqqt2/detection_loss‚2.466897690106952 0loss/SRqqt4/detection_loss‚4.379549521592356 % _timestamp‚1.7885108251884365e+09 /loss/FRt1/assignment_loss‚6.684897028166672 2loss/SRqqt2/assignment_loss‚2.0977947503801375 .loss/FBt2/detection_loss‚5.117720376700163 /loss/FBt4/detection_loss‚1.3594708223602958 epoch‚0 /loss/FRt4/assignment_loss‚7.463057092257908 2loss/SRqqt3/assignment_loss‚3.2939510999187345 /loss/FRt2/detection_loss‚2.6227320394030325 .loss/FBt1/detection_loss‚3.997778764792851 /loss/FBt3/detection_loss‚3.0703716972857364 _runtime‚ 450.658564846 1loss/SRqqt1/detection_loss‚2.4750383988259332 %loss/total_loss‚78.52874538488302 trainer/global_step‚399 /loss/FRt2/assignment_loss‚7.104960609365393 /loss/FRt3/assignment_loss‚7.226696628512758 1loss/FBt4/assignment_loss‚0.31277779335266515 _step‚31 1loss/SRqqt1/assignment_loss‚2.233849759257975 0loss/FBt2/assignment_loss‚0.5220301935842144 1loss/FBt3/assignment_loss‚0.35012939091724704 /loss/FRt3/detection_loss‚1.9615579645967844 1loss/SRqqt3/detection_loss‚1.9594904541157914QzD»VËÐ /loss/FRt4/assignment_loss‚7.463057092257908 2loss/SRqqt3/assignment_loss‚3.2939510999187345 /loss/FRt2/detection_loss‚2.6227320394030325 .loss/FBt1/detection_loss‚3.997778764792851 /loss/FBt3/detection_loss‚3.0703716972857364 _runtime‚ 450.658564846 1loss/SRqqt1/detection_loss‚2.4750383988259332 %loss/total_loss‚78.52874538488302 trainer/global_step‚399 /loss/FRt2/assignment_loss‚7.104960609365393 /loss/FRt3/assignment_loss‚7.226696628512758 1loss/FBt4/assignment_loss‚0.31277779335266515 _step‚31 1loss/SRqqt1/assignment_loss‚2.233849759257975 0loss/FBt2/assignment_loss‚0.5220301935842144 1loss/FBt3/assignment_loss‚0.35012939091724704 /loss/FRt3/detection_loss‚1.9615579645967844 1loss/SRqqt3/detection_loss‚1.9594904541157914 1loss/SRqqt4/assignment_loss‚3.482253543792232 0loss/FBt1/assignment_loss‚0.4196942813068485 .loss/FRt1/detection_loss‚3.408438966192048 .loss/FRt4/detection_loss‚4.517606518098286 0loss/SRqqt2/detection_loss‚2.466897690106952 0loss/SRqqt4/detection_loss‚4.379549521592356 % _timestamp‚1.7885108251884365e+09 /loss/FRt1/assignment_loss‚6.684897028166672 2loss/SRqqt2/assignment_loss‚2.0977947503801375 .loss/FBt2/detection_loss‚5.117720376700163 /loss/FBt4/detection_loss‚1.3594708223602958 epoch‚0ÈÞá°’Ì‚(:‡ ÷„êԤϯþ gpu.0.gpu‚23.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.195816040039062* gpu.0.memoryAllocatedBytes‚ 14771093504 gpu.0.temp‚37.0 gpu.0.powerWatts‚150.724' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚37.681 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Un0ŠÍ‚(:‡ ÷„êÔ¤‚Œÿ disk.md0.in‚0 disk.rbd2.out‚0 disk.sda.in‚0) proc.memory.availableMB‚ 1424.80078125 proc.cpu.threads‚17 disk.nvme0n1.out‚ 3080.765625 disk.sda.out‚52.42919921875 disk.sda2.out‚52.42919921875 cpu‚25.100178579573118 network.sent‚601919 disk.dm-0.in‚0.078125 disk.md0p1.out‚ 50.64453125 network.recv‚242100 disk.sdb2.in‚0$ memory_percent‚91.30370616912842! proc.memory.rssMB‚ 2893.390625 disk.sdb1.in‚0 disk.md0.out‚ 50.64453125 disk.rbd2.in‚0 disk.dm-0.out‚ 4578.27734375 disk.sdb.in‚0* proc.memory.percent‚17.659854888916016) disk./.usagePercent‚75.77838051170433 disk.sdb2.out‚52.42919921875 disk.sda1.in‚0 disk.sdb.out‚52.42919921875" disk.nvme1n1.out‚ 1497.51171875% disk./.usageGB‚2710.2019386291504 disk.nvme0n1.in‚ 0.02734375 disk.rbd0.out‚0 disk.md0p1.in‚0 disk.sda2.in‚0 disk.nvme1n1.in‚ 0.05078125 disk.rbd0.in‚0 disk.sda1.out‚0 disk.sdb1.out‚0H’QƓ΂(:ˆ †…êÔê«þý gpu.0.gpu‚51.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.195816040039062* gpu.0.memoryAllocatedBytes‚ 14771093504 gpu.0.temp‚38.0 gpu.0.powerWatts‚167.774' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚41.9435 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ì0æyüÏ‚(:ñ †…êÔéÅÿ( proc.memory.availableMB‚ 1424.9296875 disk.sda.in‚0 disk.nvme1n1.out‚ 1555.453125 disk.dm-0.out‚ 4828.3359375 disk.sda2.out‚53.83056640625 disk.md0.out‚52 disk.md0p1.in‚0 disk.nvme0n1.in‚ 0.02734375* proc.memory.percent‚17.659854888916016 proc.cpu.threads‚17 disk.sda.out‚53.83056640625 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.rbd2.in‚0 disk.sda1.in‚0 network.recv‚247445 disk.sdb1.in‚0 disk.rbd2.out‚0 disk.sda1.out‚0 disk.dm-0.in‚0.078125 disk.sdb.out‚53.83056640625) disk./.usagePercent‚75.78100180094175 disk.sdb1.out‚0 disk.sdb2.out‚53.83056640625$ memory_percent‚91.30291938781738! proc.memory.rssMB‚ 2893.390625! disk.nvme0n1.out‚ 3272.8828125% disk./.usageGB‚2710.2956886291504 disk.md0.in‚0 disk.sdb2.in‚0 disk.md0p1.out‚52 disk.sdb.in‚0 cpu‚25.09765555577985 network.sent‚641128 disk.sda2.in‚0 disk.nvme1n1.in‚ 0.05078125×…ó¨“Ђ(:ˆ •…êÔÜÇØÿ gpu.0.gpu‚54.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.195816040039062* gpu.0.memoryAllocatedBytes‚ 14771093504 gpu.0.temp‚37.0 gpu.0.powerWatts‚116.306' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚29.0765 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0’[ýÑ‚(:ý •…êÔ’¡ disk.sda1.in‚0 disk.sda1.out‚0 disk.md0.out‚ 53.86328125 disk.sdb.out‚ 55.755859375 disk.sda2.in‚0 disk.nvme1n1.in‚ 0.05078125' proc.memory.availableMB‚ 1425.015625 disk.sdb2.out‚ 55.755859375 disk.dm-0.in‚0.078125 disk.sda.in‚0 disk.sda2.out‚ 55.755859375 disk.rbd0.in‚0 disk.sdb1.in‚0 network.recv‚251179 disk.rbd2.in‚0 disk.sda.out‚ 55.755859375 disk.rbd0.out‚0* proc.memory.percent‚17.659902572631836" disk.nvme0n1.out‚ 3330.84765625" disk.nvme1n1.out‚ 1615.66796875 network.sent‚646785 disk.sdb2.in‚0$ memory_percent‚91.30239486694336 cpu‚25.09610782513603% disk./.usageGB‚2710.4206886291504 disk.dm-0.out‚ 4946.515625) disk./.usagePercent‚75.78449685325832 disk.md0.in‚0 disk.rbd2.out‚0 disk.nvme0n1.in‚ 0.02734375 disk.sdb.in‚0 disk.sdb1.out‚0" proc.memory.rssMB‚ 2893.3984375 proc.cpu.threads‚17 disk.md0p1.in‚0 disk.md0p1.out‚ 53.86328125Ä«u0Ò+ _wandbruntime‚507  _runtime‚507Ò4~>ÆÓÀ _runtime‚ 507.895740713 _step‚32 ' lr-AdamW/pg1‚0.00031853396987762783 ' lr-AdamW/pg2‚0.00031853396987762783 trainer/global_step‚449 % _timestamp‚1.7885108824251912e+09¾ÿ®WÊÔÄ _runtime‚ 507.895740713 _step‚32 ' lr-AdamW/pg1‚0.00031853396987762783 ' lr-AdamW/pg2‚0.00031853396987762783 trainer/global_step‚449 % _timestamp‚1.7885108824251912e+09 ¾+µ°0Õ+ _wandbruntime‚507  _runtime‚507c†rhÈ'ÖÂO WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚5575255 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 396436480 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚35611263933324 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚11111153 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 898640896 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30433 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚590 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 499558894076 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚8149657 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚20075909 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚8964754 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1174999693 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚36359570257920 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30433 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚40213972537344 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 223335424 % _timestamp‚1.7885108824265199e+09 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24590 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚563 _runtime‚ 507.89742298 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1154 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1170 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30188 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24592 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚20051317 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30188 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14218690560 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1170 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚11111153 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914443232 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24592 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 498661490204 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _step‚33 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚40213948597248 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚35612178376556 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1176280064 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13954824192 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚8964754 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚40735915338752 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13954824192 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 500533418496 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510207117 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚35112619482480 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24590 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 916643840 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1180696576 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚8964756 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚36360486901760 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚11086563 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1170 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚563 trainer/global_step‚449Š¦¨2Ì'×ÆO D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24592 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚20051317 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30188 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14218690560 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1170 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚11111153 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914443232 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24592 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 498661490204 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _step‚33 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚40213948597248 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚35612178376556 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1176280064 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13954824192 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚8964754 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚40735915338752 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13954824192 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 500533418496 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510207117 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚35112619482480 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24590 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 916643840 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1180696576 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚8964756 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚36360486901760 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚11086563 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1170 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚563 trainer/global_step‚449 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚5575255 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 396436480 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚35611263933324 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚11111153 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 898640896 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30433 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚590 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 499558894076 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚8149657 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚20075909 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚8964754 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1174999693 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚36359570257920 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30433 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚40213972537344 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 223335424 % _timestamp‚1.7885108824265199e+09 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24590 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚563 _runtime‚ 507.89742298 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1154 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1170 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30188!5g¶)0Ø+ _wandbruntime‚509  _runtime‚509 q¨ ÙÍM XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_enQÆ{êCd/allocated_bytes.all.current‚ 916657152 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚593 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1180696576 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚563 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13954824192 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 898654208 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚11135770 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚40303267817984 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 502544876032 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 _runtime‚ 509.032794896 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚8984654 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚11111154 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1180696576 _step‚34 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 501646221824 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚35938667134976 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30188 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 898654208 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1154 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚20120424 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚40303291758080 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14218690560 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1170 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 502544876032 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚35190582537248 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 ODeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510207117 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚563 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30433 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24616 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚8167108 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 499770151522 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1180696576 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5587631 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5587629 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 916657152 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚20120424 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚36441230013952 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 396436480 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚35190599576608 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚563 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1179_ø{×&ÚÑM QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚8984654 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚11111154 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1180696576 _step‚34 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 501646221824 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚35938667134976 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30188 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 898654208 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1154 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚20120424 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚40303291758080 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14218690560 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1170 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 502544876032 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚35190582537248 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 ODeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510207117 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚563 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30433 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24616 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚8167108 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 499770151522 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1180696576 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5587631 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5587629 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 916657152 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚20120424 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚36441230013952 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 396436480 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚35190599576608 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚563 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1179 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 916657152 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚593 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1180696576 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚563 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13954824192 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 898654208 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚11135770 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚40303267817984 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 502544876032 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 _runtime‚ 509.032794896"šz=’0Û+ _wandbruntime‚509  _runtime‚509²nYèKÜÅ /loss/FRt2/assignment_loss‚5.300766318591673 1loss/SRqqt2/detection_loss‚1.9618912009398142 0loss/SRqqt4/detection_loss‚4.285067167775384 $loss/total_loss‚76.8892552608294 trainer/global_step‚449 _step‚35 2loss/SRqqt4/assignment_loss‚3.2761024935492156 /loss/SRqqt1/detection_loss‚3.16378535495864 .loss/FBt2/detection_loss‚4.272460010051727 _runtime‚ 509.037913139 1loss/SRqqt1/assignment_loss‚2.091128244333797 1loss/FBt2/assignment_loss‚0.43347176231385676 /loss/FBt3/detection_loss‚3.5380809049371087 $ _timestamp‚1.788510883563761e+09 /loss/FRt1/assignment_loss‚6.510364770889282 /loss/FRt3/assignment_loss‚7.230818221818155 1loss/SRqqt3/assignment_loss‚2.946970160802205 0loss/FBt1/assignment_loss‚0.5140133991464443 /loss/FRt1/detection_loss‚3.0057611763477325 .loss/FRt3/detection_loss‚2.078159552011917 1loss/SRqqt3/detection_loss‚2.6222732825259705 0loss/FBt4/assignment_loss‚0.4493924004358372 .loss/FBt4/detection_loss‚2.338615880137539 /loss/FRt4/assignment_loss‚6.603645634651184 1loss/SRqqt2/assignment_loss‚2.575868448416392 0loss/FBt3/assignment_loss‚0.2813555058315008 /loss/FRt2/detection_loss‚2.1101061643949195 -loss/FRt4/detection_loss‚3.22326897457242 .loss/FBt1/detection_loss‚6.075888231396675 epoch‚0䊛OÝÉ /loss/FRt1/detection_loss‚3.0057611763477325 .loss/FRt3/detection_loss‚2.078159552011917 1loss/SRqqt3/detection_loss‚2.6222732825259705 0loss/FBt4/assignment_loss‚0.4493924004358372 .loss/FBt4/detection_loss‚2.338615880137539 /loss/FRt4/assignment_loss‚6.603645634651184 1loss/SRqqt2/assignment_loss‚2.575868448416392 0loss/FBt3/assignment_loss‚0.2813555058315008 /loss/FRt2/detection_loss‚2.1101061643949195 -loss/FRt4/detection_loss‚3.22326897457242 .loss/FBt1/detection_loss‚6.075888231396675 epoch‚0 /loss/FRt2/assignment_loss‚5.300766318591673 1loss/SRqqt2/detection_loss‚1.9618912009398142 0loss/SRqqt4/detection_loss‚4.285067167775384 $loss/total_loss‚76.8892552608294 trainer/global_step‚449 _step‚35 2loss/SRqqt4/assignment_loss‚3.2761024935492156 /loss/SRqqt1/detection_loss‚3.16378535495864 .loss/FBt2/detection_loss‚4.272460010051727 _runtime‚ 509.037913139 1loss/SRqqt1/assignment_loss‚2.091128244333797 1loss/FBt2/assignment_loss‚0.43347176231385676 /loss/FBt3/detection_loss‚3.5380809049371087 $ _timestamp‚1.788510883563761e+09 /loss/FRt1/assignment_loss‚6.510364770889282 /loss/FRt3/assignment_loss‚7.230818221818155 1loss/SRqqt3/assignment_loss‚2.946970160802205 0loss/FBt1/assignment_loss‚0.5140133991464443#G|®†”Þ‚(:‰ ¤…êÔêßÕþ gpu.0.gpu‚55.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.195816040039062* gpu.0.memoryAllocatedBytes‚ 14771093504 gpu.0.temp‚38.0 gpu.0.powerWatts‚168.715' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.17875 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ŽG]ß‚(:… ¤…êÔ±‚éÿ network.recv‚256043 disk.sdb2.out‚ 57.6318359375 disk.dm-0.in‚0.078125 disk.md0p1.in‚0 disk.sdb1.in‚0 disk.sdb1.out‚0* proc.memory.percent‚17.659902572631836! disk.nvme1n1.out‚ 1671.5234375 disk.sda1.in‚0 disk.dm-0.out‚ 5102.45703125 disk.nvme0n1.in‚ 0.02734375" disk.nvme0n1.out‚ 3444.30859375 disk.sda2.in‚0) proc.memory.availableMB‚ 1424.77734375) disk./.usagePercent‚75.79151543625352 disk.sdb.out‚ 57.6318359375 disk.nvme1n1.in‚ 0.05078125 disk.md0p1.out‚ 55.67578125 disk.rbd0.in‚0" proc.memory.rssMB‚ 2893.3984375 cpu‚25.094957415363584 disk.md0.out‚ 55.67578125% disk./.usageGB‚2710.6717071533203 disk.sda1.out‚0 disk.sdb.in‚0 disk.rbd2.out‚0$ memory_percent‚91.30384922027588 network.sent‚653068 disk.sda.in‚0 disk.sda.out‚ 57.6318359375 disk.sda2.out‚ 57.6318359375 disk.rbd0.out‚0 disk.md0.in‚0 disk.sdb2.in‚0 disk.rbd2.in‚0 proc.cpu.threads‚17å[´’à‚(:‡ ³…êÔ»è¯ÿ gpu.0.gpu‚54.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.195816040039062* gpu.0.memoryAllocatedBytes‚ 14771093504 gpu.0.temp‚38.0 gpu.0.powerWatts‚173.344' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.336 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ç:ýá‚(:ƒ ³…êÔÏÿ‚€ disk.rbd0.out‚0" disk.nvme1n1.out‚ 1758.26953125 disk.nvme0n1.in‚ 0.02734375 disk.md0p1.in‚0 network.recv‚261015 disk.sda2.in‚0 disk.md0.in‚0 disk.rbd2.out‚0 disk.dm-0.in‚0.078125 disk.sdb.out‚ 59.2236328125 cpu‚25.09280278390471 disk.sdb1.out‚0 disk.sda.in‚0 disk.rbd0.in‚0 disk.md0.out‚ 57.20703125 disk.sdb2.in‚0 disk.sda1.out‚0 disk.dm-0.out‚ 5289.19921875 disk.sdb.in‚0* proc.memory.percent‚17.659902572631836 proc.cpu.threads‚17 disk.sdb1.in‚0) disk./.usagePercent‚75.79148695789144 disk.rbd2.in‚0 disk.md0p1.out‚ 57.20703125) proc.memory.availableMB‚ 1425.19140625! disk.nvme0n1.out‚ 3530.9296875 network.sent‚704393 disk.sda2.out‚ 59.2236328125 disk.sdb2.out‚ 59.2236328125% disk./.usageGB‚2710.6706886291504 disk.sda.out‚ 59.2236328125 disk.nvme1n1.in‚ 0.05078125# memory_percent‚91.3013219833374" proc.memory.rssMB‚ 2893.3984375 disk.sda1.in‚0ö™Æ¨”â‚(:‰ Â…êÔ‘äþ gpu.0.gpu‚34.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.195816040039062* gpu.0.memoryAllocatedBytes‚ 14771093504 gpu.0.temp‚38.0 gpu.0.powerWatts‚168.383' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.09575 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0µŸ9˜ã‚(:ý Â…êÔ¦«†€ disk.rbd2.out‚0 network.sent‚708732 disk.sdb2.out‚ 60.7509765625 disk.sda.in‚0 disk.nvme0n1.in‚ 0.02734375 disk.sdb.out‚ 60.7509765625 disk.sda.out‚ 60.7509765625 network.recv‚265798 disk.dm-0.out‚ 5516.98046875 disk.sda2.out‚ 60.7509765625 disk.rbd2.in‚0 disk.rbd0.out‚0 disk.sdb1.in‚0 disk.md0.out‚ 58.671875" proc.memory.rssMB‚ 2893.3984375 cpu‚25.091665902710577 proc.cpu.threads‚17! disk.nvme0n1.out‚ 3643.0546875 disk.nvme1n1.in‚ 0.05078125" disk.nvme1n1.out‚ 1873.92578125 disk.sdb1.out‚0 disk.md0.in‚0$ memory_percent‚91.30439758300781* proc.memory.percent‚17.659902572631836) disk./.usagePercent‚75.79148695789144 disk.md0p1.in‚0 disk.md0p1.out‚ 58.671875 disk.sdb.in‚0% proc.memory.availableMB‚ 1424.6875% disk./.usageGB‚2710.6706886291504 disk.sdb2.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.in‚0 disk.rbd0.in‚0 disk.dm-0.in‚0.078125&Ïüä‚(:ƒ Ñ…êÔþÒ¯ÿ disk.rbd0.out‚0" proc.memory.rssMB‚ 2893.3984375 cpu‚25.089788075700614 disk.rbd2.out‚0 disk.sdb.in‚0) proc.memory.availableMB‚ 1424.43359375* proc.memory.percent‚17.659902572631836 proc.cpu.threads‚17" disk.nvme0n1.out‚ 3780.96484375 disk.sda2.in‚0 disk.rbd0.in‚0 disk.rbd2.in‚0 disk.dm-0.in‚0.078125 disk.nvme1n1.in‚ 0.05078125 disk.sdb1.in‚0 disk.sdb1.out‚0$ memory_percent‚91.30594730377197# disk./.usageGB‚2710.53263092041! disk.nvme1n1.out‚ 1931.8046875 disk.md0.out‚ 60.19140625 disk.sdb2.in‚0 disk.md0p1.out‚ 60.19140625 disk.nvme0n1.in‚ 0.02734375 disk.sdb.out‚ 62.3388671875 network.sent‚714388 disk.sdb2.out‚ 62.3388671875 disk.sda1.in‚0 network.recv‚269521) disk./.usagePercent‚75.78762680657343 disk.md0.in‚0 disk.dm-0.out‚ 5712.76953125 disk.sda.in‚0 disk.sda2.out‚ 62.3388671875 disk.sda1.out‚0 disk.md0p1.in‚0 disk.sda.out‚ 62.3388671875LRçÅ’å‚(:‡ Ñ…êÔÛç¤ÿ gpu.0.gpu‚55.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.195816040039062* gpu.0.memoryAllocatedBytes‚ 14771093504 gpu.0.temp‚38.0 gpu.0.powerWatts‚164.372' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚41.093 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0n v€0æ+ _wandbruntime‚564  _runtime‚564(ÕëbÄç¾ _step‚36 ' lr-AdamW/pg1‚0.00035400545872814315 ' lr-AdamW/pg2‚0.00035400545872814315 trainer/global_step‚499 $ _timestamp‚1.788510938957297e+09 _runtime‚ 564.42761652 83¾Èè ' lr-AdamW/pg1‚0.00035400545872814315 ' lr-AdamW/pg2‚0.00035400545872814315 trainer/global_step‚499 $ _timestamp‚1.788510938957297e+09 _runtime‚ 564.42761652 _step‚36$¾AB¸0é+ _wandbruntime‚564  _runtime‚564Qì¤K2êÂO H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24592 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30433 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30188 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2860779 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 227529728 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 246679040 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914443232 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1170 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚9959656 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 556187998720 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚9959656 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚621 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 396436480 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 580145079296 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚39011607149520 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 557086639616 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 556187998720 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1180696576 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1170 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1176280064 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1179 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚563 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚12319465 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚39840551617024 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6194055 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚39566629740064 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14218690560 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚40396739615744 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 554108147312 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚6194053 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 579917549568 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897403872 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 898640896 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14218690560 trainer/global_step‚499 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚22279119 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2861398 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510207117 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30188 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚619 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚39011624188880 _step‚37 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24592 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚45259827183616 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1170 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚39840551617024 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚40396739615744 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1176280064 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14218690560 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1180696576 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13954824192 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 898640896 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 916643840 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 251469824 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚44679933574144 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚39565715296832 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚9959654 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1170 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607½Š7ûÌ'ëÆO XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 898640896 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14218690560 trainer/global_step‚499 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚22279119 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2861398 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510207117 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30188 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚619 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚39011624188880 _step‚37 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24592 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚45259827183616 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1170 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚39840551617024 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚40396739615744 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1176280064 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14218690560 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1180696576 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13954824192 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 898640896 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 916643840 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 251469824 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚44679933574144 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚39565715296832 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚9959654 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1170 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24592 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30433 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30188 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2860779 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 227529728 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 246679040 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914443232 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1170 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚9959656 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 556187998720 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚9959656 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚621 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 396436480 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 580145079296 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚39011607149520 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 557086639616 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 556187998720 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1180696576 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1170 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1176280064 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1179 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚563 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚12319465 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚39840551617024 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6194055 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚39566629740064 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14218690560 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚40396739615744 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 554108147312 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚6194053 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 579917549568 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897403872%ÀÑK0ì+ _wandbruntime‚565  _runtime‚565ü!¥Ó&íÍM KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _step‚38 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13954824192 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 558199456256 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1154 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚39645718495718 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510207117 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚563 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 898654208 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 555216808630 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚9979552 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚597 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚9979554 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14218690560 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 916657152 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚39089587243648 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1170 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 898654208 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2867167 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1174999693 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24618 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30433 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 557300802048 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚22323608 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚12368672 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚22348226 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 581082584576 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30433 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚9979554 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚9072999 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14218690560 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚39920181912576 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚45350311439360 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 221224960 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚39089604283008 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚12344056 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 556114212710 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚563 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚22348226 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 557300802048 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 245165056 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13954824192 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚44769228854784 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 246679040 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1170 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚9979552 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2866570 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚6206431 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 581303809536 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1180696576 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚40478399371776 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚599 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚39644804052278 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚563 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚499 % _timestamp‚1.7885109399116275e+09 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1179 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚44769252794880 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚39920181912576 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1176280064 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897404080 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚39920199915520 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1180696576 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1180696576 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚40478399371776 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚12368672 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚9073598 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6206429 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510207117 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚563 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 898654208 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 555216808630 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚9979552 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚597 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚9979554 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14218690560 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 916657152 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚39089587243648 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1170 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 898654208 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2867167 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1174999693 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24618 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30433 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 557300802048 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚22323608 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚12368672 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚22348226 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 581082584576 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30433 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚9979554 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚9072999 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14218690560 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚39920181912576 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚45350311439360 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 221224960 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚39089604283008 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚12344056 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 556114212710 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚563 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚22348226 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 557300802048 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 245165056 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13954824192 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚44769228854784 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 246679040 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1170 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚9979552 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2866570 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚6206431 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 581303809536 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1180696576 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚40478399371776 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚599 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚39644804052278 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚563 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚499 % _timestamp‚1.7885109399116275e+09 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1179&oÌÀ´0ï+ _wandbruntime‚565  _runtime‚565€'j!PðÊ 1loss/SRqqt1/detection_loss‚3.3910948148795534 %loss/total_loss‚84.86775998553966 /loss/FRt3/assignment_loss‚6.310965231486729 1loss/SRqqt3/assignment_loss‚3.131937405892781 0loss/FBt1/assignment_loss‚0.2946900548897568 /loss/FRt1/assignment_loss‚5.931566622949416 2loss/SRqqt4/assignment_loss‚3.1880111789703367 .loss/FBt2/detection_loss‚9.845086562633515 /loss/FBt4/detection_loss‚2.4322025633416047 epoch‚0 1loss/SRqqt1/assignment_loss‚2.698938007865633 0loss/FBt2/assignment_loss‚1.3174640053534858 % _timestamp‚1.7885109399140687e+09 /loss/FRt2/assignment_loss‚4.450334876775742 .loss/FRt4/assignment_loss‚5.92733561424982 1loss/SRqqt2/assignment_loss‚2.675642917030736 1loss/SRqqt3/detection_loss‚2.1104090677146052 .loss/FBt1/detection_loss‚3.847965725204524 trainer/global_step‚499 _step‚39 /loss/FRt2/detection_loss‚2.8009934835135937 1loss/SRqqt2/detection_loss‚1.8884140914796215 .loss/FBt3/detection_loss‚7.958417047153821 1loss/FBt4/assignment_loss‚0.33670988884193287 .loss/FRt3/detection_loss‚3.252910617561567 /loss/FRt4/detection_loss‚2.8879686225028265 1loss/SRqqt4/detection_loss‚4.1970488691329955 _runtime‚ 565.384190691 0loss/FBt3/assignment_loss‚0.9727262475273826 .loss/FRt1/detection_loss‚3.018926468587691C€èTñÎ /loss/FRt1/assignment_loss‚5.931566622949416 2loss/SRqqt4/assignment_loss‚3.1880111789703367 .loss/FBt2/detection_loss‚9.845086562633515 /loss/FBt4/detection_loss‚2.4322025633416047 epoch‚0 1loss/SRqqt1/assignment_loss‚2.698938007865633 0loss/FBt2/assignment_loss‚1.3174640053534858 % _timestamp‚1.7885109399140687e+09 /loss/FRt2/assignment_loss‚4.450334876775742 .loss/FRt4/assignment_loss‚5.92733561424982 1loss/SRqqt2/assignment_loss‚2.675642917030736 1loss/SRqqt3/detection_loss‚2.1104090677146052 .loss/FBt1/detection_loss‚3.847965725204524 trainer/global_step‚499 _step‚39 /loss/FRt2/detection_loss‚2.8009934835135937 1loss/SRqqt2/detection_loss‚1.8884140914796215 .loss/FBt3/detection_loss‚7.958417047153821 1loss/FBt4/assignment_loss‚0.33670988884193287 .loss/FRt3/detection_loss‚3.252910617561567 /loss/FRt4/detection_loss‚2.8879686225028265 1loss/SRqqt4/detection_loss‚4.1970488691329955 _runtime‚ 565.384190691 0loss/FBt3/assignment_loss‚0.9727262475273826 .loss/FRt1/detection_loss‚3.018926468587691 1loss/SRqqt1/detection_loss‚3.3910948148795534 %loss/total_loss‚84.86775998553966 /loss/FRt3/assignment_loss‚6.310965231486729 1loss/SRqqt3/assignment_loss‚3.131937405892781 0loss/FBt1/assignment_loss‚0.2946900548897568'7·?”ò‚(:‰ à…êÔ„õ­ÿ gpu.0.gpu‚50.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.195816040039062* gpu.0.memoryAllocatedBytes‚ 14771093504 gpu.0.temp‚37.0 gpu.0.powerWatts‚174.903' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.72575 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0·'5 ó‚(:€ à…êÔ°ªÃÿ disk.sdb.out‚ 64.2861328125 network.recv‚274439" disk.nvme0n1.out‚ 3881.15234375 disk.sdb.in‚0 disk.sda2.in‚0" disk.nvme1n1.out‚ 1989.73828125 disk.sdb2.out‚ 64.2861328125 disk.dm-0.out‚ 5870.890625 disk.sda.in‚0 disk.rbd0.in‚0 disk.md0.in‚0* proc.memory.percent‚17.659902572631836% disk./.usageGB‚2710.1154251098633 disk.sdb2.in‚0 disk.md0p1.in‚0 network.sent‚720724 disk.sda.out‚ 64.2861328125 disk.sda2.out‚ 64.2861328125 disk.sdb1.in‚0 disk.md0.out‚ 62.0703125 disk.rbd2.in‚0 cpu‚25.088352473495885 disk.nvme1n1.in‚ 0.05078125 disk.sdb1.out‚0 disk.rbd2.out‚0" proc.memory.rssMB‚ 2893.3984375 disk.sda1.out‚0 proc.cpu.threads‚17) disk./.usagePercent‚75.77596155749634 disk.sda1.in‚0 disk.dm-0.in‚0.078125 disk.md0p1.out‚ 62.0703125 disk.rbd0.out‚0$ memory_percent‚91.30620956420898' proc.memory.availableMB‚ 1424.390625 disk.nvme0n1.in‚ 0.027343753%·ž’ô‚(:‡ ï…êÔãÚìÿ gpu.0.gpu‚24.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.195816040039062* gpu.0.memoryAllocatedBytes‚ 14771093504 gpu.0.temp‚38.0 gpu.0.powerWatts‚173.676' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.419 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0KqäŽõ‚(:‡ ï…êÔœ‡­€ disk.sdb.out‚66.33349609375! disk.nvme1n1.out‚ 2076.4765625 disk.sda1.in‚0 disk.sdb.in‚0 disk.rbd0.in‚0 disk.md0.out‚ 64.05859375 disk.dm-0.out‚ 6041.73828125 disk.md0p1.out‚ 64.05859375" disk.nvme0n1.out‚ 3965.26171875 disk.sda.in‚0 disk.nvme1n1.in‚ 0.05078125 disk.rbd2.out‚0 proc.cpu.threads‚17 disk.sda1.out‚0 disk.dm-0.in‚0.078125 disk.sda2.in‚0 network.recv‚279097$ memory_percent‚91.30215644836426" proc.memory.rssMB‚ 2893.3984375 network.sent‚771691) disk./.usagePercent‚75.77595089144313 disk.sdb2.in‚0 disk.md0p1.in‚0 disk.sda.out‚66.33349609375 disk.sda2.out‚66.33349609375 disk.rbd0.out‚0 disk.sdb1.out‚0 disk.nvme0n1.in‚ 0.02734375 disk.sdb1.in‚0 disk.md0.in‚0 disk.rbd2.in‚0( proc.memory.availableMB‚ 1425.0546875* proc.memory.percent‚17.659902572631836 cpu‚25.08651963511361% disk./.usageGB‚2710.1150436401367 disk.sdb2.out‚66.33349609375÷îös”ö‚(:‰ þ…êÔ¦•Þÿ gpu.0.gpu‚43.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.195816040039062* gpu.0.memoryAllocatedBytes‚ 14771093504 gpu.0.temp‚37.0 gpu.0.powerWatts‚158.137' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚39.53425 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‡L—÷‚(:ý þ…êÔûɧ disk.rbd2.in‚0 disk.sda1.in‚0" proc.memory.rssMB‚ 2893.3984375* proc.memory.percent‚17.659902572631836% disk./.usageGB‚2709.5875205993652 disk.sda.in‚0 disk.rbd0.out‚0 disk.sdb1.in‚0! disk.nvme0n1.out‚ 3965.3671875 cpu‚25.08516438263953 proc.cpu.threads‚17 disk.sdb2.in‚0 disk.sda1.out‚0 disk.md0.out‚ 65.46484375 disk.sdb.in‚0 disk.sda.out‚ 67.7880859375 disk.sda2.out‚ 67.7880859375! disk.nvme1n1.out‚ 2077.0234375 disk.dm-0.in‚0.078125% proc.memory.availableMB‚ 1424.9375 network.sent‚778070) disk./.usagePercent‚75.76120112643761 disk.sda2.in‚0 disk.rbd0.in‚0 disk.nvme1n1.in‚ 0.05078125 disk.md0.in‚0 disk.rbd2.out‚0 disk.md0p1.in‚0 disk.sdb1.out‚0 disk.sdb2.out‚ 67.7880859375 disk.dm-0.out‚ 6042.390625 disk.nvme0n1.in‚ 0.02734375 disk.md0p1.out‚ 65.46484375$ memory_percent‚91.30287170410156 network.recv‚283998 disk.sdb.out‚ 67.7880859375X»X”ø‚(:‰ †êÔ±°—ÿ gpu.0.gpu‚11.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.195816040039062* gpu.0.memoryAllocatedBytes‚ 14771093504 gpu.0.temp‚38.0 gpu.0.powerWatts‚175.235' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.80875 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚09ŒV«ù‚(:… †êÔº¹Ùÿ disk.sdb2.in‚0 disk.dm-0.out‚ 6091.78515625 disk.sdb1.out‚0 network.recv‚287680 disk.sda1.in‚0 disk.rbd0.out‚0 disk.nvme0n1.out‚ 3966.296875 disk.sda.in‚0 disk.sda2.out‚69.80908203125 disk.md0.in‚0" proc.memory.rssMB‚ 2893.3984375 disk.dm-0.in‚0.078125 disk.md0p1.in‚0 disk.md0p1.out‚ 67.4296875 disk.rbd2.out‚0 disk.nvme0n1.in‚ 0.02734375 disk.sdb.in‚0 disk.sdb.out‚69.80908203125 disk.sda2.in‚0 network.sent‚781676 disk.rbd0.in‚0 disk.sdb1.in‚0 disk.md0.out‚ 67.4296875) proc.memory.availableMB‚ 1424.80859375 cpu‚25.083690947834643 proc.cpu.threads‚17 disk.rbd2.in‚0 disk.sdb2.out‚69.80908203125 disk.sda1.out‚0 disk.nvme1n1.in‚ 0.05078125* proc.memory.percent‚17.659902572631836 disk.sda.out‚69.80908203125" disk.nvme1n1.out‚ 2125.48828125# memory_percent‚91.3036584854126) disk./.usagePercent‚75.76071624765859% disk./.usageGB‚2709.57017898559577ž o0ú+ _wandbruntime‚616  _runtime‚616ñ|ƒ!Åû¿ ' lr-AdamW/pg1‚0.00038947694757865853 ' lr-AdamW/pg2‚0.00038947694757865853 trainer/global_step‚549 $ _timestamp‚1.788510991308577e+09 _runtime‚ 616.778910449 _step‚40à¿Æ¯Éüà trainer/global_step‚549 $ _timestamp‚1.788510991308577e+09 _runtime‚ 616.778910449 _step‚40 ' lr-AdamW/pg1‚0.00038947694757865853 ' lr-AdamW/pg2‚0.00038947694757865853(M¨ª0ý+ _wandbruntime‚616  _runtime‚616`Ï‘*Í'þÇO ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚49784051321856 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚44433908973568 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 223335424 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚49145870670848 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚10954554 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3148449 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 611842578944 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1180696576 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚9961302 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1174999693 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 916643840 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 398533632 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 637933375488 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1180696576 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24590 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚44434825617408 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1170 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚42910611855920 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚24531513 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚586 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914443232 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1170 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚10954556 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚13576957 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚10954554 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30188 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚584 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚13552367 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897403872 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚10954556 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 610452208292 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 898640896 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 611842578944 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24590 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6812855 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚49145894610944 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚43521081103572 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 898640896 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚44433908973568 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1176280064 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚49783804046336 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14218690560 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚563 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1176280064 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚13576957 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13954824192 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚42910628895280 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚43822066394624 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 246679040 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14218690560 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚563 trainer/global_step‚549 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24592 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30188 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚44434825617408 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 916643840 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚6812853 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1155 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚43520166660340 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚43822066394624˼YÈÑ'ÿËO TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚24531513 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚586 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914443232 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1170 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚10954556 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚13576957 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚10954554 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30188 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚584 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚13552367 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897403872 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚10954556 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 610452208292 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 898640896 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 611842578944 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24590 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6812855 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚49145894610944 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚43521081103572 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 898640896 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚44433908973568 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1176280064 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚49783804046336 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14218690560 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚563 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1176280064 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚13576957 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13954824192 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚42910628895280 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚43822066394624 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 246679040 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14218690560 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚563 trainer/global_step‚549 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24592 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30188 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚44434825617408 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 916643840 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚6812853 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1155 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚43520166660340 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚43822066394624 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚49784051321856 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚44433908973568 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 223335424 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚49145870670848 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚10954554 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3148449 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 611842578944 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1180696576 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚9961302 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1174999693 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 916643840 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 398533632 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 637933375488 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1180696576 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24590 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚44434825617408 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1170 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚42910611855920)Jôã0€+ _wandbruntime‚617  _runtime‚617Lf†G ÐM K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚24576028 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚13601574 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1180696576 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24618 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1180 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚42988608989408 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 916657152 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚49235189891584 ODeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1180696576 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚24576028 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 916657152 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 610663465738 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14218690560 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 223322112 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚9979460 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13954824192 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 247262208 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1176280064 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚10974454 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚13576958 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3154231 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1170 _runtime‚ 617.74059065 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚43901714693120 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚9980063 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914443440 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1170 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1170 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚563 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 612955382272 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚43901696690176 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30433 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚49874533188096 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 398533632 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510207117 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚563 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚44515568729600 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 898654208 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14218690560 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1176280064 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14218690560 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚49235213831680 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1174999693 trainer/global_step‚549 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13954824192 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚24551410 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚10974452 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 639319356416 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 611560869818 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30433 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚13576958 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚42988591950048 % _timestamp‚1.7885109922693577e+09 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚44514652072448 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚13601574 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1180696576 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚24576028 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 916657152 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 610663465738 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14218690560 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 223322112 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚9979460 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13954824192 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 247262208 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1176280064 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚10974454 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚13576958 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3154231 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1170 _runtime‚ 617.74059065 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚43901714693120 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚9980063 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914443440 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1170 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1170 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚563 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 612955382272 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚43901696690176 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30433 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚49874533188096 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 398533632 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510207117 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚563 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚44515568729600 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 898654208 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14218690560 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1176280064 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14218690560 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚49235213831680 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1174999693 trainer/global_step‚549 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13954824192 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚24551410 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚10974452 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 639319356416 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 611560869818 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30433 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚13576958 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚42988591950048 % _timestamp‚1.7885109922693577e+09 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚44514652072448 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚13601574 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚24576028 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚13601574 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1180696576 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24618 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984*åéH0ƒ+ _wandbruntime‚617  _runtime‚617-¾O„É .loss/FBt3/detection_loss‚9.253235645592213 %loss/total_loss‚75.59999104911309 0loss/FBt4/assignment_loss‚1.0762341306121512 /loss/FRt1/detection_loss‚2.9533216694990796 1loss/SRqqt3/detection_loss‚2.4412207241963455 2loss/SRqqt4/assignment_loss‚3.2207897592473915 .loss/FBt2/detection_loss‚2.006028898605486 1loss/SRqqt1/detection_loss‚3.3476389574593512 _step‚43 1loss/SRqqt3/assignment_loss‚2.733596748775906 epoch‚0 trainer/global_step‚549 % _timestamp‚1.7885109922719033e+09 /loss/FRt2/detection_loss‚2.1068034699329963 /loss/FRt2/assignment_loss‚4.398706417817336 /loss/FRt3/assignment_loss‚5.926301414316351 /loss/FRt4/assignment_loss‚5.718819693515175 1loss/SRqqt1/assignment_loss‚2.156848637194469 -loss/FRt3/detection_loss‚2.93142731487751 /loss/FBt1/detection_loss‚2.5931782359662265 2loss/SRqqt2/assignment_loss‚2.6682700208089885 0loss/FBt1/assignment_loss‚0.2206379874993519 0loss/FBt3/assignment_loss‚0.7139225116698071 0loss/SRqqt2/detection_loss‚1.019408376001328 1loss/SRqqt4/detection_loss‚3.3095298122476646 .loss/FBt4/detection_loss‚5.330409717153419 /loss/FRt1/assignment_loss‚6.170772361755371 0loss/FBt2/assignment_loss‚0.3613280234316024 .loss/FRt4/detection_loss‚2.941560520937568 _runtime‚ 617.742023727¡Û0ùS…Í .loss/FBt4/detection_loss‚5.330409717153419 /loss/FRt1/assignment_loss‚6.170772361755371 0loss/FBt2/assignment_loss‚0.3613280234316024 .loss/FRt4/detection_loss‚2.941560520937568 _runtime‚ 617.742023727 .loss/FBt3/detection_loss‚9.253235645592213 %loss/total_loss‚75.59999104911309 0loss/FBt4/assignment_loss‚1.0762341306121512 /loss/FRt1/detection_loss‚2.9533216694990796 1loss/SRqqt3/detection_loss‚2.4412207241963455 2loss/SRqqt4/assignment_loss‚3.2207897592473915 .loss/FBt2/detection_loss‚2.006028898605486 1loss/SRqqt1/detection_loss‚3.3476389574593512 _step‚43 1loss/SRqqt3/assignment_loss‚2.733596748775906 epoch‚0 trainer/global_step‚549 % _timestamp‚1.7885109922719033e+09 /loss/FRt2/detection_loss‚2.1068034699329963 /loss/FRt2/assignment_loss‚4.398706417817336 /loss/FRt3/assignment_loss‚5.926301414316351 /loss/FRt4/assignment_loss‚5.718819693515175 1loss/SRqqt1/assignment_loss‚2.156848637194469 -loss/FRt3/detection_loss‚2.93142731487751 /loss/FBt1/detection_loss‚2.5931782359662265 2loss/SRqqt2/assignment_loss‚2.6682700208089885 0loss/FBt1/assignment_loss‚0.2206379874993519 0loss/FBt3/assignment_loss‚0.7139225116698071 0loss/SRqqt2/detection_loss‚1.019408376001328 1loss/SRqqt4/detection_loss‚3.3095298122476646+Þ v3“†‚(:ˆ œ†êÔé‹Öþ gpu.0.gpu‚31.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.195816040039062* gpu.0.memoryAllocatedBytes‚ 14771093504 gpu.0.temp‚37.0 gpu.0.powerWatts‚115.966' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.9915 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ìbÇ> ‡‚(:þ œ†êÔѶÁÿ disk.sda.out‚ 71.3447265625 disk.sdb1.in‚0 disk.md0.in‚0( disk./.usagePercent‚75.7612578698407' proc.memory.availableMB‚ 1424.484375* proc.memory.percent‚17.659950256347656 proc.cpu.threads‚17 disk.sdb.out‚ 71.3447265625 disk.nvme1n1.in‚ 0.05078125" disk.nvme1n1.out‚ 2267.80859375 disk.sdb2.in‚0 disk.md0p1.in‚0 disk.sdb.in‚0 disk.sda2.out‚ 71.3447265625 disk.sdb1.out‚0 disk.sdb2.out‚ 71.3447265625 disk.dm-0.in‚0.078125 disk.sda.in‚0 disk.rbd0.in‚0 disk.rbd2.out‚0 disk.dm-0.out‚ 6234.2578125 network.sent‚787640 disk.sda2.in‚0 disk.rbd0.out‚0 disk.md0.out‚ 68.8984375 disk.sda1.out‚0 proc.memory.rssMB‚ 2893.40625 cpu‚25.082173308612276 network.recv‚292294% disk./.usageGB‚2709.5895500183105 disk.md0p1.out‚ 68.8984375 disk.nvme0n1.in‚ 0.02734375" disk.nvme0n1.out‚ 3966.44921875 disk.rbd2.in‚0 disk.sda1.in‚0$ memory_percent‚91.30563735961914û™¦“ˆ‚(:ˆ «†êÔ˜ÿ gpu.0.gpu‚52.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.195816040039062* gpu.0.memoryAllocatedBytes‚ 14771093504 gpu.0.temp‚38.0 gpu.0.powerWatts‚169.942' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.4855 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0õÔ⌠‰‚(:ÿ «†êÔØÌÝÿ disk.sdb.in‚0 disk.sdb.out‚ 72.9560546875 disk.sda.in‚0 disk.sdb1.out‚0 disk.sdb2.in‚0 disk.dm-0.in‚0.078125 disk.md0p1.out‚ 70.44140625 disk.nvme0n1.in‚ 0.02734375 disk.sda2.in‚0! disk.nvme1n1.out‚ 2349.2890625) proc.memory.availableMB‚ 1424.42578125* proc.memory.percent‚17.659950256347656 network.sent‚836851 disk.sda1.in‚0 proc.memory.rssMB‚ 2893.40625% disk./.usageGB‚2709.6684913635254 disk.rbd0.in‚0 disk.md0.in‚0 disk.rbd2.in‚0 proc.cpu.threads‚17 disk.sdb1.in‚0 disk.sda.out‚ 72.9560546875 disk.nvme0n1.out‚ 3966.453125 disk.sda2.out‚ 72.9560546875# memory_percent‚91.3059949874878 cpu‚25.081062378154893 network.recv‚297458 disk.rbd0.out‚0 disk.sda1.out‚0) disk./.usagePercent‚75.76346510289243 disk.dm-0.out‚ 6315.7421875 disk.md0p1.in‚0 disk.nvme1n1.in‚ 0.05078125 disk.md0.out‚ 70.44140625 disk.sdb2.out‚ 72.9560546875 disk.rbd2.out‚0÷¯”Š‚(:‰ º†êÔ‡üÿ gpu.0.gpu‚52.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.195816040039062* gpu.0.memoryAllocatedBytes‚ 14771093504 gpu.0.temp‚38.0 gpu.0.powerWatts‚153.793' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚38.44825 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0]§aû‹‚(:û º†êÔæ¨þ€ disk.dm-0.out‚ 6318.69921875 disk.sda.out‚ 74.431640625 disk.rbd2.in‚0 disk.sdb2.out‚ 74.431640625 disk.sdb.in‚0 proc.memory.rssMB‚ 2893.40625 disk.sda1.in‚0 disk.sda1.out‚0 disk.nvme1n1.in‚ 0.05078125 disk.sdb1.in‚0 disk.md0p1.in‚0 disk.md0p1.out‚ 71.85546875 disk.nvme0n1.in‚ 0.02734375 network.sent‚845291 disk.rbd0.in‚0 disk.md0.in‚0 disk.sdb2.in‚0* proc.memory.percent‚17.659950256347656 cpu‚25.07992211229176 disk.rbd0.out‚0" disk.nvme1n1.out‚ 2352.01171875 disk.nvme0n1.out‚ 3966.6875 disk.sda.in‚0$ memory_percent‚91.30752086639404) proc.memory.availableMB‚ 1424.17578125 disk.md0.out‚ 71.85546875 proc.cpu.threads‚17 disk.sdb.out‚ 74.431640625% disk./.usageGB‚2709.7646865844727 disk.sdb1.out‚0 disk.rbd2.out‚0 disk.sda2.in‚0 disk.sda2.out‚ 74.431640625 network.recv‚302334) disk./.usagePercent‚75.76615476153094 disk.dm-0.in‚0.078125@º†0Œ+ _wandbruntime‚671  _runtime‚671ˆˆEÆÀ _step‚44 ' lr-AdamW/pg1‚0.00042494843642917386 ' lr-AdamW/pg2‚0.00042494843642917386 trainer/global_step‚599 % _timestamp‚1.7885110460348656e+09 _runtime‚ 671.505199379ì9[’ÊŽÄ trainer/global_step‚599 % _timestamp‚1.7885110460348656e+09 _runtime‚ 671.505199379 _step‚44 ' lr-AdamW/pg1‚0.00042494843642917386 ' lr-AdamW/pg2‚0.00042494843642917386,ï1=0+ _wandbruntime‚671  _runtime‚6719úPMÉO ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 665898865400 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897403872 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24592 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚46809616562320 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1157 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24590 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30433 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3437391 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1174999693 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14218690560 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚7431655 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚48471078331392 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 898640896 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚14785269 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚54308214635520 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚14809859 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1182 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 398533632 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚563 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24592 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚47803581172224 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 898640896 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3438008 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30188 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 696358987776 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚563 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚10869663 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14218690560 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚53611831707648 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1170 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14218690560 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚14809859 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚47803599175168 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚7431653 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914443232 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1170 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 668395800064 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚11949454 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 696131458048 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚563 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1176280064 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚619 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚47803581172224 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 251469824 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚53611855647744 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 246679040 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1180696576 % _timestamp‚1.7885110460356016e+09 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 916643840 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13954824192 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 227529728 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚11949454 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 667497159168 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚47803599175168 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 668395800064 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24590 trainer/global_step‚599 _runtime‚ 671.506048163 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚48471994975232 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚10869044 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚47474618023848 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚48471078331392 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 916643840 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_alls}`¥‚ ocations.peak‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚47475532467080 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30433 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 667497159168 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚11949456 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1180696576 _step‚45 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚48471994975232 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 665001461528 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30188oAkÓ'‘ÍO E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚26734723 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13954824192 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚617 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1176280064 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510207117 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1170 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚26759315 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚54307963165696 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚46809633601680 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30188 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 665898865400 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897403872 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24592 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚46809616562320 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1157 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24590 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30433 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3437391 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1174999693 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14218690560 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚7431655 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚48471078331392 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 898640896 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚14785269 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚54308214635520 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚14809859 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1182 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 398533632 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚563 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24592 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚47803581172224 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 898640896 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3438008 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30188 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 696358987776 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚563 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚10869663 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14218690560 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚53611831707648 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1170 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14218690560 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚14809859 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚47803599175168 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚7431653 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914443232 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1170 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 668395800064 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚11949454 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 696131458048 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚563 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1176280064 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚619 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚47803581172224 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 251469824 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚53611855647744 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 246679040 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1180696576 % _timestamp‚1.7885110460356016e+09 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 916643840 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13954824192 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 227529728 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚11949454 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 667497159168 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚47803599175168 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 668395800064 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24590 trainer/global_step‚599 _runtime‚ 671.506048163 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚48471994975232 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚10869044 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚47474618023848 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚48471078331392 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 916643840 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚47475532467080 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30433 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 667497159168 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚11949456 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1180696576 _step‚45 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚48471994975232 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 665001461528 IDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚10887179 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24616 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 916657152 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 trainer/global_step‚599 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚14834476 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚617 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 666110122846 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚11969354 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1180696576 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24618 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 668609962496 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1182 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3443150 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚47883229470720 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 229613568 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚53701150928384 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510207117 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚48552738087424 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚11969352 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7444031 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13954824192 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30433 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 898654208 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚26803830 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚47883211467776 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1176280064 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30433 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3443767 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1157 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 398533632 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚46887596656448 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 697289568256 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚47883211467776 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1176280064 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚11969354 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1180696576 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚46887613695808 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1170 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚563 O8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚47553706779294 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 669508616704 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚10887798 % _timestamp‚1.7885110471369157e+09 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚48552738087424 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚47554621222734 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 _runtime‚ 672.608134818ÙŸrÝ&”×M VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚47883211467776 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1176280064 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30433 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3443767 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1157 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 398533632 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚46887596656448 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 697289568256 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚47883211467776 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1176280064 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚11969354 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1180696576 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚46887613695808 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1170 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚563 O8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚47553706779294 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 669508616704 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚10887798 % _timestamp‚1.7885110471369157e+09 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚48552738087424 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚47554621222734 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 _runtime‚ 672.608134818 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚48551821430272 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚54398694050304 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚26779212 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24616 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 253553664 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914443440 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚26779212 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 668609962496 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚10887179 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24616 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 916657152 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 trainer/global_step‚599 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚14834476 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚617 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 666110122846 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚11969354 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1180696576 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24618 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 668609962496 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1182 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3443150 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚47883229470720 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 229613568 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚53701150928384 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510207117 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚48552738087424 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚11969352 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7444031 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13954824192 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30433 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 898654208 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚26803830 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677.‰>“ 0•+ _wandbruntime‚672  _runtime‚672ü«Ðd–Ì /loss/FRt4/assignment_loss‚5.827889038168865 0loss/FBt3/assignment_loss‚1.43262982368=Wºî46924 /loss/FRt3/detection_loss‚2.4472818241474474 0loss/SRqqt3/detection_loss‚1.621510331371897 2loss/SRqqt2/assignment_loss‚2.9463576329381844 1loss/SRqqt3/assignment_loss‚2.617580153440174 0loss/SRqqt4/detection_loss‚3.907020138369666 trainer/global_step‚599 /loss/FRt2/assignment_loss‚3.695382501528813 /loss/FRt3/assignment_loss‚5.436021530881841 2loss/SRqqt4/assignment_loss‚2.7391986780696445 1loss/FBt2/assignment_loss‚0.29534855888895517 0loss/FRt1/assignment_loss‚6.0236153364181515 .loss/FBt2/detection_loss‚1.327414909664255 $loss/total_loss‚81.6592090704255 epoch‚0 _runtime‚ 672.610181768 _step‚47 2loss/SRqqt1/assignment_loss‚2.7659266873409876 0loss/FBt1/assignment_loss‚0.5799547266700068 0loss/FBt4/assignment_loss‚0.7748539488055997 .loss/FRt1/detection_loss‚2.972021061182022 /loss/FBt1/detection_loss‚2.8347785726592347 0loss/SRqqt1/detection_loss‚3.708095023506566 .loss/FBt3/detection_loss‚19.44929313659668 % _timestamp‚1.7885110471394842e+09 /loss/FRt2/detection_loss‚1.9735597491264343 /loss/FRt4/detection_loss‚2.4954018968602885 1loss/SRqqt2/detection_loss‚0.7271484720079522 /loss/FBt4/detection_loss‚3.0609253380971495®¿¼ÕV—Ð trainer/global_step‚599 /loss/FRt2/assignment_loss‚3.695382501528813 /loss/FRt3/assignment_loss‚5.436021530881841 2loss/SRqqt4/assignment_loss‚2.7391986780696445 1loss/FBt2/assignment_loss‚0.29534855888895517 0loss/FRt1/assignment_loss‚6.0236153364181515 .loss/FBt2/detection_loss‚1.327414909664255 $loss/total_loss‚81.6592090704255 epoch‚0 _runtime‚ 672.610181768 _step‚47 2loss/SRqqt1/assignment_loss‚2.7659266873409876 0loss/FBt1/assignment_loss‚0.5799547266700068 0loss/FBt4/assignment_loss‚0.7748539488055997 .loss/FRt1/detection_loss‚2.972021061182022 /loss/FBt1/detection_loss‚2.8347785726592347 0loss/SRqqt1/detection_loss‚3.708095023506566 .loss/FBt3/detection_loss‚19.44929313659668 % _timestamp‚1.7885110471394842e+09 /loss/FRt2/detection_loss‚1.9735597491264343 /loss/FRt4/detection_loss‚2.4954018968602885 1loss/SRqqt2/detection_loss‚0.7271484720079522 /loss/FBt4/detection_loss‚3.0609253380971495 /loss/FRt4/assignment_loss‚5.827889038168865 0loss/FBt3/assignment_loss‚1.4326298236846924 /loss/FRt3/detection_loss‚2.4472818241474474 0loss/SRqqt3/detection_loss‚1.621510331371897 2loss/SRqqt2/assignment_loss‚2.9463576329381844 1loss/SRqqt3/assignment_loss‚2.617580153440174 0loss/SRqqt4/detection_loss‚3.907020138369666/Lô~I’˜‚(:‡ ɆêÔ­Îãý gpu.0.gpu‚27.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.195816040039062* gpu.0.memoryAllocatedBytes‚ 14771093504 gpu.0.temp‚37.0 gpu.0.powerWatts‚124.044' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚31.011 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0M’ù™‚(:î ɆêÔ˜Õ¿€ disk.sdb2.in‚0 disk.sdb.out‚ 76.3837890625 network.sent‚861619 disk.md0p1.out‚73.75 disk.nvme1n1.in‚ 0.05078125 disk.sda1.out‚0 disk.sda2.in‚0' proc.memory.availableMB‚ 1424.703125) disk./.usagePercent‚75.77404188123921% disk./.usageGB‚2710.0467681884766 disk.sda.in‚0 network.recv‚306129 disk.sda2.out‚ 76.3837890625 disk.rbd0.in‚0 disk.sdb1.out‚0" disk.nvme0n1.out‚ 3966.69140625 disk.sda1.in‚0 disk.md0p1.in‚0 disk.nvme0n1.in‚ 0.02734375 disk.rbd0.out‚0$ memory_percent‚91.30430221557617 cpu‚25.0789611103767 disk.dm-0.in‚0.078125 disk.dm-0.out‚ 6455.44140625 disk.sdb.in‚0 disk.sdb1.in‚0 disk.rbd2.in‚0 proc.memory.rssMB‚ 2893.40625* proc.memory.percent‚17.659950256347656 disk.sdb2.out‚ 76.3837890625 disk.sda.out‚ 76.3837890625 disk.nvme1n1.out‚2488.75 disk.md0.in‚0 disk.md0.out‚73.75 disk.rbd2.out‚0 proc.cpu.threads‚17Orz7”š‚(:‰ ؆êԚу€ gpu.0.gpu‚45.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.195816040039062* gpu.0.memoryAllocatedBytes‚ 14771093504 gpu.0.temp‚38.0 gpu.0.powerWatts‚152.567' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚38.14175 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚09󬛂(:ƒ ؆êÔ›ï disk.sda.in‚0 disk.dm-0.out‚ 6712.04296875" disk.nvme1n1.out‚ 2745.20703125 disk.sdb2.in‚0 cpu‚25.077958670357543 disk.md0.in‚0 disk.rbd2.in‚0 disk.nvme0n1.in‚ 0.02734375 disk.sda2.in‚0 disk.sdb1.out‚0 disk.md0.out‚ 75.328125 disk.rbd2.out‚0 disk.sda1.out‚0 disk.md0p1.in‚0 proc.memory.rssMB‚ 2893.40625 network.recv‚311105 disk.sdb2.out‚78.02783203125! disk.nvme0n1.out‚ 3966.8359375 disk.sdb.out‚78.02783203125 disk.nvme1n1.in‚ 0.05078125) proc.memory.availableMB‚ 1424.93359375* proc.memory.percent‚17.659950256347656 disk.md0p1.out‚ 75.328125 disk.sda2.out‚78.02783203125 disk.rbd0.in‚0 disk.rbd0.out‚0 proc.cpu.threads‚17) disk./.usagePercent‚75.77752114779702% disk./.usageGB‚2710.1712036132812 disk.dm-0.in‚0.078125 disk.sda.out‚78.02783203125 disk.sdb1.in‚0$ memory_percent‚91.30289554595947 network.sent‚899808 disk.sda1.in‚0 disk.sdb.in‚0$ù<«”œ‚(:‰ ç†êÔ·¡Úý gpu.0.gpu‚29.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.195816040039062* gpu.0.memoryAllocatedBytes‚ 14771093504 gpu.0.temp‚37.0 gpu.0.powerWatts‚126.369' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚31.59225 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Än’‚(:ƒ ç†êÔÿ disk.sda.in‚0 proc.memory.rssMB‚ 2893.40625 disk.rbd0.out‚0 network.recv‚315113( disk./.usagePercent‚75.7915293021227$ disk./.usageGB‚2710.672203063965 disk.sdb1.out‚0$ memory_percent‚91.30513668060303 disk.sdb2.in‚0 disk.md0p1.out‚ 76.8203125 disk.nvme0n1.in‚ 0.02734375" disk.nvme0n1.out‚ 3966.83984375 disk.sda.out‚79.57470703125 disk.sda2.out‚79.57470703125 disk.rbd0.in‚0* proc.memory.percent‚17.659950256347656 disk.rbd2.out‚0 cpu‚25.077183735325498 proc.cpu.threads‚17 network.sent‚903409 disk.md0.out‚ 76.8203125 disk.sdb2.out‚79.57470703125 disk.sda1.in‚0 disk.md0p1.in‚0 disk.nvme1n1.in‚ 0.05078125 disk.sda1.out‚0 disk.rbd2.in‚0 disk.dm-0.out‚ 6850.45703125 disk.sdb.out‚79.57470703125! disk.nvme1n1.out‚ 2883.6171875 disk.sdb1.in‚0 disk.md0.in‚0 disk.sda2.in‚0) proc.memory.availableMB‚ 1424.56640625 disk.dm-0.in‚0.078125 disk.sdb.in‚0!í¯`’ž‚(:‡ ö†êÔõú‘þ gpu.0.gpu‚34.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.195816040039062* gpu.0.memoryAllocatedBytes‚ 14771093504 gpu.0.temp‚37.0 gpu.0.powerWatts‚114.408' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.602 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Éö'蟂(:† ö†êÔçÄœÿ$ memory_percent‚91.30842685699463 proc.memory.rssMB‚ 2893.40625 network.recv‚319525 disk.nvme1n1.in‚ 0.05078125! disk.nvme1n1.out‚ 3041.5859375 disk.sdb1.in‚0 disk.nvme0n1.in‚ 0.02734375 network.sent‚909630 disk.sdb2.in‚0 disk.sdb.out‚81.66943359375 disk.sda.in‚0 disk.sda2.out‚81.66943359375 disk.sdb2.out‚81.66943359375 disk.rbd2.out‚0 disk.rbd0.out‚0) proc.memory.availableMB‚ 1424.02734375 cpu‚25.076295667137693 disk.sda1.in‚0 disk.sda1.out‚0 disk.md0p1.in‚0 disk.md0p1.out‚ 78.83984375 disk.rbd0.in‚0$ disk./.usageGB‚2710.671184539795 disk.rbd2.in‚0 disk.dm-0.out‚ 7008.51171875 disk.md0.out‚ 78.83984375* proc.memory.percent‚17.659950256347656 disk.md0.in‚0" disk.nvme0n1.out‚ 3966.92578125 disk.sdb.in‚0 proc.cpu.threads‚17 disk.sda2.in‚0) disk./.usagePercent‚75.79150082376063 disk.sdb1.out‚0 disk.dm-0.in‚0.078125 disk.sda.out‚81.66943359375ÀQû0 + _wandbruntime‚727  _runtime‚727"y˜Ä¡¾ & lr-AdamW/pg2‚0.0004604199252796892 trainer/global_step‚649 % _timestamp‚1.7885111023010736e+09 _runtime‚ 727.771527272 _step‚48 & lr-AdamW/pg1‚0.0004604199252796892A4¾@Ȣ _runtime‚ 727.771527272 _step‚48 & lr-AdamW/pg1‚0.0004604199252796892 & lr-AdamW/pg2‚0.0004604199252796892 trainer/global_step‚649 % _timestamp‚1.7885111023010736e+090oLŸ@0£+ _wandbruntime‚727  _runtime‚727Ô+î‰Ï'¤ÉO XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897403872 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚612 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30188 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚11776741 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚58831985992192 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13954824192 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚16018171 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 253566976 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1180696576 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1170 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚563 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚12944354 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1182 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24592 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚28962525 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚52508247689216 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚16042761 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚28987117 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚52509164333056 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚16042761 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚58077816684544 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚52509164333056 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 754422874624 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 _runtime‚ 727.772940258 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 723151739392 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1170 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚563 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24590 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24590 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3726286 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3725674 % _timestamp‚1.7885111023023353e+09 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 916643840 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30188 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 724050380288 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 898640896 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24592 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1180696576 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚52508247689216 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚51785113952768 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚50708638308080îRíÓ'¥ÍO VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1176280064 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1157 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1170 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 724050380288 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 398533632 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 898640896 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚58077792744448 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1176280064 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 721345522508 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1180696576 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 916643840 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24592 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1180696576 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚52508247689216 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚51785113952768 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚50708638308080 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897403872 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚612 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30188 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚11776741 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚58831985992192 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13954824192 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚16018171 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 253566976 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1180696576 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1170 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚563 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚12944354 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1182 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24592 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚28962525 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚52508247689216 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚16042761 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚28987117 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚52509164333056 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚16042761 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚58077816684544 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚52509164333056 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 754422874624 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 _runtime‚ 727.772940258 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 723151739392 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1170 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚563 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24590 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24590 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3726286 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3725674 % _timestamp‚1.7885111023023353e+09 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 916643840 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30188 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 724050380288 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 898640896 IDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 916657152 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510207117 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30433 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚11794198 trainer/global_step‚649 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚605 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3731972 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1180696576 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚12964252 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚51864726245376 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚16067378 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚58922719280128 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 898654208 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚29007014 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8062831 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 755370538496 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914443440 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚563 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚29031632 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1170 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 916657152 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14218690560 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚563 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚12964254 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13954824192 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 236776448 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚58167111965184 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 755583374848 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 _step‚50 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚58167135905280 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚12964254 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8062829 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚51509072586242 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897404080 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1180696576 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1170 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚12964252 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 722454184034 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚563 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30433 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1180696576 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚51508158142802 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚50786618402208 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚51864744248320 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚52588990788096 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚11794803 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚6077Ó(Ý&¨×M J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚12964252 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚51864726245376 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚16067378 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚58922719280128 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 898654208 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚29007014 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8062831 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 755370538496 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914443440 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚563 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚29031632 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1170 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 916657152 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14218690560 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚563 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚12964254 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13954824192 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 236776448 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚58167111965184 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 755583374848 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 _step‚50 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚58167135905280 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚12964254 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8062829 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚51509072586242 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897404080 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1180696576 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1170 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚12964252 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 722454184034 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚563 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30433 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1180696576 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚51508158142802 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚50786618402208 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚51864744248320 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚52588990788096 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚11794803 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14218690560 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 725163196928 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14218690560 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ODeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 916657152 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510207117 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30433 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚11794198 trainer/global_step‚649 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚605 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3731972 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 11806965762P†g¼0©+ _wandbruntime‚728  _runtime‚728–X«QªË .loss/FBt3/detection_loss‚4.772689302762349 /loss/FRt3/assignment_loss‚5.613466280698776 0loss/FBt2/assignment_loss‚0.2725961988187897 0loss/FBt4/assignment_loss‚0.4066833431725157 /loss/FRt4/assignment_loss‚6.205169571770562 2loss/SRqqt1/assignment_loss‚2.6265000965860157 1loss/SRqqt2/assignment_loss‚2.612361065312928 .loss/FBt1/detection_loss‚2.952070782892406 /loss/FBt4/detection_loss‚2.5480455780760494 /loss/FRt1/assignment_loss‚6.206002495505593 0loss/FRt2/assignment_loss‚3.3209589454862805 2loss/SRqqt3/assignment_loss‚2.4611290259794756 2loss/SRqqt4/assignment_loss‚2.7895763715108237 0loss/FBt1/assignment_loss‚0.5067880373192679 /loss/FRt1/detection_loss‚2.8754931417378513 /loss/FRt2/detection_loss‚1.8422409900360637 .loss/FRt4/detection_loss‚2.416362492574586 1loss/SRqqt3/detection_loss‚1.5768947470368762 0loss/SRqqt4/detection_loss‚3.100189330180486 .loss/FBt2/detection_loss‚1.733375179448298 1loss/SRqqt2/detection_loss‚0.9698700988117386 %loss/total_loss‚64.84336741728296 % _timestamp‚1.7885111033828714e+09 /loss/SRqqt1/detection_loss‚3.39294132259157 _step‚51 epoch‚0 trainer/global_step‚649 0loss/FBt3/assignment_loss‚0.9243370195229849 /loss/FRt3/detection_loss‚2.7176259994506835 _runtime‚ 728.853032102¥˜µpU«Ï trainer/global_step‚649 0loss/FBt3/assignment_loss‚0.9243370195229849 /loss/FRt3/detection_loss‚2.7176259994506835 _runtime‚ 728.853032102 .loss/FBt3/detection_loss‚4.772689302762349 /loss/FRt3/assignment_loss‚5.613466280698776 0loss/FBt2/assignment_loss‚0.2725961988187897 0loss/FBt4/assignment_loss‚0.4066833431725157 /loss/FRt4/assignment_loss‚6.205169571770562 2loss/SRqqt1/assignment_loss‚2.6265000965860157 1loss/SRqqt2/assignment_loss‚2.612361065312928 .loss/FBt1/detection_loss‚2.952070782892406 /loss/FBt4/detection_loss‚2.5480455780760494 /loss/FRt1/assignment_loss‚6.206002495505593 0loss/FRt2/assignment_loss‚3.3209589454862805 2loss/SRqqt3/assignment_loss‚2.4611290259794756 2loss/SRqqt4/assignment_loss‚2.7895763715108237 0loss/FBt1/assignment_loss‚0.5067880373192679 /loss/FRt1/detection_loss‚2.8754931417378513 /loss/FRt2/detection_loss‚1.8422409900360637 .loss/FRt4/detection_loss‚2.416362492574586 1loss/SRqqt3/detection_loss‚1.5768947470368762 0loss/SRqqt4/detection_loss‚3.100189330180486 .loss/FBt2/detection_loss‚1.733375179448298 1loss/SRqqt2/detection_loss‚0.9698700988117386 %loss/total_loss‚64.84336741728296 % _timestamp‚1.7885111033828714e+09 /loss/SRqqt1/detection_loss‚3.39294132259157 _step‚51 epoch‚03ùŒó’¬‚(:‡ …‡êÔÁ÷˜€ gpu.0.gpu‚48.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.195816040039062* gpu.0.memoryAllocatedBytes‚ 14771093504 gpu.0.temp‚38.0 gpu.0.powerWatts‚153.176' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚38.294 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÈAKØ­‚(:ü …‡êÔÇâÐ disk.sdb1.in‚0 disk.dm-0.in‚0.078125 disk.sdb.in‚0" disk.nvme1n1.out‚ 3247.71484375 disk.md0.in‚0 disk.rbd2.in‚0 disk.md0p1.in‚0 proc.memory.rssMB‚ 2893.40625 network.recv‚323224 disk.rbd2.out‚0 disk.sdb2.out‚ 83.224609375 disk.sda1.in‚0 disk.sda1.out‚0* proc.memory.percent‚17.659950256347656 disk.dm-0.out‚ 7214.72265625 disk.sda.out‚ 83.224609375 disk.nvme1n1.in‚ 0.05078125 disk.md0p1.out‚ 80.33203125 disk.sdb1.out‚0 disk.sdb2.in‚0 network.sent‚914793 disk.sda2.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.md0.out‚ 80.33203125' proc.memory.availableMB‚ 1424.703125 proc.cpu.threads‚17 disk.sda2.out‚ 83.224609375$ memory_percent‚91.30430221557617 cpu‚25.075611558437455) disk./.usagePercent‚75.79150082376063$ disk./.usageGB‚2710.671184539795 disk.nvme0n1.in‚ 0.02734375! disk.nvme0n1.out‚ 3967.0078125 disk.sdb.out‚ 83.224609375 disk.sda.in‚0cXã4’®‚(:‡ ”‡êԭζÿ gpu.0.gpu‚47.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.195816040039062* gpu.0.memoryAllocatedBytes‚ 14771093504 gpu.0.temp‚38.0 gpu.0.powerWatts‚162.204' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚40.551 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0“Ó§,¯‚(:„ ”‡êÔ׿‰ disk.sda.in‚0 disk.rbd0.in‚0 disk.nvme1n1.in‚ 0.05078125 proc.cpu.threads‚17 network.recv‚328165 disk.rbd2.in‚0 disk.rbd2.out‚0 disk.sdb2.out‚84.66064453125 disk.sda1.out‚0 disk.sda2.out‚84.66064453125 disk.rbd0.out‚0 disk.sdb1.in‚0 disk.md0.out‚ 81.70703125 disk.sdb2.in‚0 cpu‚25.07450869023074$ memory_percent‚91.30182266235352' proc.memory.availableMB‚ 1425.109375# proc.memory.rssMB‚ 2893.41015625$ disk./.usageGB‚2710.671184539795" disk.nvme1n1.out‚ 3436.55078125 disk.sdb1.out‚0 disk.md0.in‚0* proc.memory.percent‚17.659974098205566 disk.sda2.in‚0 disk.sda1.in‚0 network.sent‚965904) disk./.usagePercent‚75.79150082376063 disk.sdb.in‚0 disk.dm-0.in‚0.078125 disk.dm-0.out‚ 7403.83203125 disk.md0p1.out‚ 81.70703125 disk.nvme0n1.in‚ 0.02734375 disk.sda.out‚84.66064453125 disk.md0p1.in‚0 disk.nvme0n1.out‚ 3967.28125 disk.sdb.out‚84.66064453125äW*0”°‚(:‰ £‡êÔ³ô¢þ gpu.0.gpu‚13.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.195816040039062* gpu.0.memoryAllocatedBytes‚ 14771093504 gpu.0.temp‚38.0 gpu.0.powerWatts‚173.953' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.48825 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¼p ±‚(:ý £‡êÔÌ¿õÿ disk.rbd2.in‚0 disk.sda1.in‚0" disk.nvme1n1.out‚ 3623.33203125 disk.rbd2.out‚0 disk.sda1.out‚0 disk.sdb.in‚0 disk.sda.in‚0 disk.sda.out‚ 86.62890625$ memory_percent‚91.30594730377197 proc.cpu.threads‚17 disk.nvme1n1.in‚ 0.05078125 disk.sdb1.out‚0 disk.sdb2.out‚ 86.62890625" disk.nvme0n1.out‚ 3967.28515625 disk.sda2.in‚0 disk.sda2.out‚ 86.62890625 disk.dm-0.out‚ 7590.6171875) proc.memory.availableMB‚ 1424.43359375$ disk./.usageGB‚2710.671184539795 disk.sdb2.in‚0 disk.md0p1.in‚0 disk.sdb.out‚ 86.62890625 cpu‚25.073804156975605 disk.md0.out‚ 83.62109375 disk.dm-0.in‚0.078125 disk.md0p1.out‚ 83.62109375 network.sent‚969648 disk.nvme0n1.in‚ 0.02734375 disk.rbd0.in‚0 disk.rbd0.out‚0# proc.memory.rssMB‚ 2893.41015625* proc.memory.percent‚17.659974098205566 network.recv‚332213) disk./.usagePercent‚75.79150082376063 disk.sdb1.in‚0 disk.md0.in‚0®V“²‚(:ˆ ²‡êÔ©•³ÿ gpu.0.gpu‚34.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.195816040039062* gpu.0.memoryAllocatedBytes‚ 14771093504 gpu.0.temp‚38.0 gpu.0.powerWatts‚160.922' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚40.2305 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ød=. ³‚(:€ ²‡êÔãèèÿ disk.nvme0n1.in‚ 0.02734375" disk.nvme1n1.out‚ 3683.61328125 disk.rbd2.in‚0 proc.cpu.threads‚17 disk.md0p1.out‚ 84.9453125# proc.memory.rssMB‚ 2893.41015625 network.recv‚337064 disk.sdb2.in‚0 disk.sdb2.out‚88.02294921875 disk.dm-0.out‚ 7650.94140625 disk.rbd0.in‚0% proc.memory.availableMB‚ 1424.8125* proc.memory.percent‚17.659974098205566 cpu‚25.0730560334695 disk.sda1.in‚0 disk.rbd0.out‚0 disk.rbd2.out‚0 disk.sda2.out‚88.02294921875 disk.sdb1.in‚0 disk.md0.in‚0$ memory_percent‚91.30363464355469 network.sent‚975827) disk./.usagePercent‚75.77675297864468 disk.sdb.in‚0 disk.sda.in‚0 disk.md0.out‚ 84.9453125$ disk./.usageGB‚2710.143730163574 disk.sda1.out‚0 disk.dm-0.in‚0.078125 disk.md0p1.in‚0 disk.sda.out‚88.02294921875 disk.sdb1.out‚0 disk.nvme0n1.out‚ 3967.328125 disk.sdb.out‚88.02294921875 disk.sda2.in‚0 disk.nvme1n1.in‚ 0.05078125OR[ç0´+ _wandbruntime‚783  _runtime‚783Nïqĵ¾ & lr-AdamW/pg1‚0.0004958914141302046 & lr-AdamW/pg2‚0.0004958914141302046 trainer/global_step‚699 % _timestamp‚1.7885111585196714e+09 _runtime‚ 783.990134809 _step‚52Þ ‘õȶ _runtime‚ 783.990134809 _step‚52 & lr-AdamW/pg1‚0.0004958914141302046 & lr-AdamW/pg2‚0.0004958914141302046 trainer/global_step‚699 % _timestamp‚1.7885111585196714e+094àOÐ\0·+ _wandbruntime‚783  _runtime‚783Vå¨Ë¤¸ÉO SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1176280064 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897403872 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24590 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24592 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8669255 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 246679040 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1189 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚618 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 776792179616 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1180696576 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚31214919 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚12684757 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 812350340608 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1174999693 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4015502 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1170 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 779704960512 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 775894775744 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 916643840 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚55766628730368 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1180696576 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1180696576 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30433 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚13939256 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚56545417047040 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30433 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30188 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 916643840 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1176280064 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚13939256 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚563 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚56545417047040 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚63356363980288 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚55384435194096 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚54607643014480 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1170 \IDeviceStatsMonitor.onŽ9Í+ _train_batch_start/active_bytes.large_pool.allocated‚55766628730368 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8669253 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1164 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 778806319616 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚17275663 % _timestamp‚1.7885111585208135e+09 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14218690560 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14218690560 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚12684137 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _runtime‚ 783.991391744 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚563 _step‚53 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚55766610727424 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚17251073 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚13939254 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13954824192 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 898640896 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4014884 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914443232 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1170 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1170 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 778806319616 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚620 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚54607625975120 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0ÛÈ9Ó'¹ÍO KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24590 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24592 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8669255 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 246679040 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1189 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚618 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 776792179616 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1180696576 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚31214919 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚12684757 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 812350340608 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1174999693 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4015502 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1170 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 779704960512 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 775894775744 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 916643840 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚55766628730368 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1180696576 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1180696576 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30433 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚13939256 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚56545417047040 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30433 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30188 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 916643840 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1176280064 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚13939256 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚563 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚56545417047040 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚63356363980288 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚55384435194096 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚54607643014480 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1170 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚55766628730368 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8669253 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1164 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 778806319616 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚17275663 % _timestamp‚1.7885111585208135e+09 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14218690560 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14218690560 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚12684137 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _runtime‚ 783.991391744 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚563 _step‚53 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚55766610727424 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚17251073 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚13939254 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13954824192 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 898640896 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4014884 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914443232 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1170 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1170 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 778806319616 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚620 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚54607625975120 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1176280064 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897403872 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚05Ü–©Â0º+ _wandbruntime‚785  _runtime‚785|‰§Ù&»ÓM J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚17275664 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 777003437062 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1180696576 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚563 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚13959154 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914443440 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1170 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚31259434 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚31259434 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1189 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1174999693 trainer/global_step‚699 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24616 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510207117 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1170 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚63446587451392 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚13959152 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13954824192 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1170 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30433 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4021165 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 813737771520 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚55846241022976 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚634 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚632 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚17275664 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 247262208 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 916657152 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 223322112 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚54685606069248 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚563 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8681631 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 779919122944 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚56627076803072 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1164 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 246679040 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚55462609506310 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1176280064 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24616 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 777900841142 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚56627076803072 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚56626160145920 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚17300280 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7885111595598617e+09 _step‚54 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 813514449408 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚54685623108608 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚55846241022976 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 898654208 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚62633073001984 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14218690560 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 398533632 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14218690560 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚12702162 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24618 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 _runtime‚ 785.030843415 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚63446834713600¾ÝÖ$¼×M `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 813737771520 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚55846241022976 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚634 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚632 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚17275664 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 247262208 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 916657152 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 223322112 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚54685606069248 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚563 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8681631 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 779919122944 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚56627076803072 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1164 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 246679040 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚55462609506310 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1176280064 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24616 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 777900841142 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚56627076803072 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚56626160145920 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚17300280 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7885111595598617e+09 _step‚54 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 813514449408 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚54685623108608 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚55846241022976 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 898654208 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚62633073001984 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14218690560 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 398533632 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14218690560 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚12702162 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24618 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 _runtime‚ 785.030843415 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚63446834713600 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚17275664 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 777003437062 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1180696576 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚563 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚13959154 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914443440 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1170 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚31259434 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚31259434 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1189 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1174999693 trainer/global_step‚699 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24616 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510207117 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1170 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚63446587451392 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚13959152 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13954824192 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1170 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30433 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚40211656¦ ·0½+ _wandbruntime‚785  _runtime‚785lè¾=¾· .loss/FRt1/detection_loss‚3.233734351893266 0loss/SRqqt1/detection_loss‚3.104286724870855 epoch‚0 /loss/FRt4/assignment_loss‚5.798512786626816 0loss/FBt2/assignment_loss‚0.4197170859660787 loss/FBt3/assignment_loss‚0 /loss/FRt2/detection_loss‚2.0029420234940267 /loss/FBt2/detection_loss‚1.1417009626515209 -loss/FBt4/detection_loss‚2.71299037906183 .loss/FBt3/detection_loss‚2.978300094604492 %loss/total_loss‚59.86009665891538 1loss/SRqqt2/assignment_loss‚2.584832924407917 /loss/FRt4/detection_loss‚2.8260881239548326 0loss/SRqqt3/detection_loss‚1.599305204817882 _runtime‚ 785.03179987 .loss/FRt1/assignment_loss‚5.94990998506546 0loss/FBt4/assignment_loss‚0.6426220203167755 /loss/FRt3/assignment_loss‚4.747583623115833 2loss/SRqqt1/assignment_loss‚3.0903760303150523 _step‚55 % _timestamp‚1.7885111595616784e+09 1loss/SRqqt2/detection_loss‚0.8892406708264089 1loss/SRqqt4/detection_loss‚3.2843907409244113 /loss/FRt2/assignment_loss‚2.933890085870569 2loss/SRqqt3/assignment_loss‚2.6320668899095976 1loss/SRqqt4/assignment_loss‚2.484302822086546 /loss/FRt3/detection_loss‚2.2990458676448235 /loss/FBt1/detection_loss‚1.8745579418208864 0loss/FBt1/assignment_loss‚0.6296993186694951 trainer/global_step‚699wÊÔ[A¿» % _timestamp‚1.7885111595616784e+09 1loss/SRqqt2/detection_loss‚0.8892406708264089 1loss/SRqqt4/detection_loss‚3.2843907409244113 /loss/FRt2/assignment_loss‚2.933890085870569 2loss/SRqqt3/assignment_loss‚2.6320668899095976 1loss/SRqqt4/assignment_loss‚2.484302822086546 /loss/FRt3/detection_loss‚2.2990458676448235 /loss/FBt1/detection_loss‚1.8745579418208864 0loss/FBt1/assignment_loss‚0.6296993186694951 trainer/global_step‚699 .loss/FRt1/detection_loss‚3.233734351893266 0loss/SRqqt1/detection_loss‚3.104286724870855 epoch‚0 /loss/FRt4/assignment_loss‚5.798512786626816 0loss/FBt2/assignment_loss‚0.4197170859660787 loss/FBt3/assignment_loss‚0 /loss/FRt2/detection_loss‚2.0029420234940267 /loss/FBt2/detection_loss‚1.1417009626515209 -loss/FBt4/detection_loss‚2.71299037906183 .loss/FBt3/detection_loss‚2.978300094604492 %loss/total_loss‚59.86009665891538 1loss/SRqqt2/assignment_loss‚2.584832924407917 /loss/FRt4/detection_loss‚2.8260881239548326 0loss/SRqqt3/detection_loss‚1.599305204817882 _runtime‚ 785.03179987 .loss/FRt1/assignment_loss‚5.94990998506546 0loss/FBt4/assignment_loss‚0.6426220203167755 /loss/FRt3/assignment_loss‚4.747583623115833 2loss/SRqqt1/assignment_loss‚3.0903760303150523 _step‚557jÕ¶%‘À‚(:† Á‡êÔº‚×þ gpu.0.gpu‚14.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.195816040039062* gpu.0.memoryAllocatedBytes‚ 14771093504 gpu.0.temp‚36.0 gpu.0.powerWatts‚114.74' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.685 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¥¬ª Á‚(:€ Á‡êÔà…êÿ disk.sda.out‚ 89.5634765625 disk.md0p1.in‚0 disk.md0p1.out‚ 86.43359375# proc.memory.rssMB‚ 2893.41015625 disk.rbd0.out‚0 disk.nvme1n1.in‚ 0.05078125 disk.sdb1.in‚0 disk.nvme0n1.out‚ 3967.46875 disk.sdb.out‚ 89.5634765625 disk.sda.in‚0 disk.md0.in‚0 proc.cpu.threads‚17 network.sent‚980985 network.recv‚340255! disk.nvme1n1.out‚ 3982.3671875 disk.sdb2.out‚ 89.5634765625 disk.nvme0n1.in‚ 0.02734375 cpu‚25.07232985518461% disk./.usageGB‚2710.1434364318848 disk.sda2.in‚0 disk.sda2.out‚ 89.5634765625 disk.sdb1.out‚0 disk.md0.out‚ 86.43359375$ memory_percent‚91.30356311798096 disk.sdb2.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-0.in‚0.078125 disk.dm-0.out‚ 7949.8359375 disk.sdb.in‚0) proc.memory.availableMB‚ 1424.82421875* proc.memory.percent‚17.659974098205566( disk./.usagePercent‚75.7767447657837 disk.rbd0.in‚0 disk.rbd2.in‚0 disk.rbd2.out‚0w‡£”‚(:‰ ЇêÔúïéý gpu.0.gpu‚57.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.195816040039062* gpu.0.memoryAllocatedBytes‚ 14771093504 gpu.0.temp‚38.0 gpu.0.powerWatts‚174.903' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.72575 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0•E” Â(:ÿ ЇêÔ÷ñþ disk.sdb.in‚0 disk.sda.in‚0 disk.sda1.out‚0 disk.sda1.in‚0! disk.nvme0n1.out‚ 3967.5703125 disk.sda.out‚ 91.1640625 disk.rbd2.in‚0 disk.sdb2.in‚0 disk.sdb2.out‚ 91.1640625) proc.memory.availableMB‚ 1424.49609375 proc.cpu.threads‚17 disk.rbd0.in‚0 disk.rbd0.out‚0" disk.nvme1n1.out‚ 4103.40234375 disk.rbd2.out‚0 disk.dm-0.out‚ 8070.97265625$ memory_percent‚91.30556583404541* proc.memory.percent‚17.659974098205566 disk.sdb.out‚ 91.1640625 disk.sda2.out‚ 91.1640625 disk.md0.out‚ 87.98046875 cpu‚25.071389029081633 network.sent‚ 1.030196e+06 network.recv‚345309) disk./.usagePercent‚75.77148981468714$ disk./.usageGB‚2709.955493927002 disk.nvme0n1.in‚ 0.02734375 disk.sda2.in‚0 disk.md0.in‚0# proc.memory.rssMB‚ 2893.41015625 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.md0p1.in‚0 disk.md0p1.out‚ 87.98046875 disk.nvme1n1.in‚ 0.05078125 disk.dm-0.in‚0.078125½c“Ä‚(:ˆ ߇êÔ¢š©ÿ gpu.0.gpu‚40.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.195816040039062* gpu.0.memoryAllocatedBytes‚ 14771093504 gpu.0.temp‚38.0 gpu.0.powerWatts‚167.434' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚41.8585 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0÷iµÅ‚(:Ž ߇êÔïúÿ disk.sdb1.in‚0 disk.md0p1.out‚ 90.0546875" disk.nvme1n1.out‚ 4187.67578125 disk.sdb1.out‚0 disk.sda1.out‚0 proc.cpu.threads‚17 disk.sda2.in‚0 disk.nvme1n1.in‚ 0.05078125 cpu‚25.070651718171153) disk./.usagePercent‚75.76198188153275% disk./.usageGB‚2709.6154441833496" disk.nvme0n1.out‚ 4044.70703125 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.md0.out‚ 90.0546875$ memory_percent‚91.30551815032959* proc.memory.percent‚17.659974098205566 network.sent‚ 1.037999e+06 disk.rbd2.in‚0 disk.rbd2.out‚0 disk.md0p1.in‚0) proc.memory.availableMB‚ 1424.50390625 network.recv‚349310 disk.sdb.out‚93.30517578125 disk.sda.in‚0 disk.sda.out‚93.30517578125 disk.md0.in‚0 disk.sdb2.in‚0 disk.sdb2.out‚93.30517578125 disk.dm-0.out‚ 8232.3828125 disk.nvme0n1.in‚ 0.02734375 disk.sdb.in‚0# proc.memory.rssMB‚ 2893.41015625 disk.sda2.out‚93.30517578125 disk.sda1.in‚0 disk.dm-0.in‚0.078125{è~ 0Æ+ _wandbruntime‚836  _runtime‚836|™c@ÄǾ % _timestamp‚1.7885112112661111e+09 _runtime‚ 836.736475318 _step‚56 & lr-AdamW/pg1‚0.0005313629029807199 & lr-AdamW/pg2‚0.0005313629029807199 trainer/global_step‚749 ³ŒÂÈÈ trainer/global_step‚749 % _timestamp‚1.7885112112661111e+09 _runtime‚ 836.736475318 _step‚56 & lr-AdamW/pg1‚0.0005313629029807199 & lr-AdamW/pg2‚0.00053136290298071998«‰J20É+ _wandbruntime‚836  _runtime‚836žßöÏ'ÊÉO VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 834460899840 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚60583503048704 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30433 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚60582586404864 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 898640896 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 831341432852 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14218690560 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1180696576 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1170 % _timestamp‚1.7885112112668552e+09 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9288055 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30188 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 835359540736 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚59338886557604 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚18483975 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚563 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 916643840 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚612 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 870372910080 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914443232 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14218690560 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚563 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚59748143507968 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1176280064 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 835359540736 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1176280064 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚14934156 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 398533632 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚14934154 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24592 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 832238836724 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚60583503048704 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚59748143507968 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚13592257 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1170 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚60582586404864 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0¾r]Ó'ËÍO dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚67009738758144 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24590 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚59748125505024 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 916643840 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13954824192 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30433 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897403872 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 870600439808 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 246679040 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚58506630681520 _step‚57 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 898640896 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24592 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 834460899840 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30188 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚59337972114372 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚33442721 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24590 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚18483975 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9288053 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4304202 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14218690560 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚18508565 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1164 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 251469824 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚33418129 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚14934154 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4303590 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚67880339197952 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 _runtime‚ 836.737308142 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚60583503048704 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30433 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚60582586404864 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 898640896 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 831341432852 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14218690560 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1180696576 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1170 % _timestamp‚1.7885112112668552e+09 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9288055 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30188 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 835359540736 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚59338886557604 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚18483975 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚563 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 916643840 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚612 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 870372910080 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914443232 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14218690560 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚563 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚59748143507968 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1176280064 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 835359540736 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1176280064 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚14934156 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 398533632 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚14934154 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24592 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 832238836724 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚60583503048704 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚59748143507968 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚13592257 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1170 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚60582586404864 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 834460899840 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚33487236 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚619 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚13609810 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚58584627815008 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚58584610775648 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30433 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13954824192 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 871757696000 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 245165056 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚59417975313258 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14218690560 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 916657152 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 833347498250 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1180696576 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1170 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚563 trainer/global_step‚749 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510207117 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897404080 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14218690560 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13954824192 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚14954052 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚67099034038784 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914443440 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚14954054 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 898654208 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1164 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚18533182 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚67099057978880 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚33487236 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24616 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 835573703168 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚67970815674880 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30433 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 246679040 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚563 $ _timestamp‚1.788511212220048e+09 FßÜ&ÎÖM TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 835573703168 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1176280064 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚60663329503744 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1176280064 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚33487236 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚619 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚13609810 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚58584627815008 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚58584610775648 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30433 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13954824192 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 871757696000 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 245165056 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚59417975313258 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14218690560 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 916657152 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 833347498250 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1180696576 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1170 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚563 trainer/global_step‚749 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510207117 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897404080 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14218690560 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13954824192 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚14954052 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚67099034038784 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914443440 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚14954054 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 898654208 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1164 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚18533182 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚67099057978880 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚33487236 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24616 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 835573703168 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚67970815674880 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30433 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 246679040 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚563 $ _timestamp‚1.788511212220048e+09 F0Ï+ _wandbruntime‚837  _runtime‚837È`°NÐÈ trainer/global_step‚749 /loss/FRt1/assignment_loss‚5.880095620950063 1loss/SRqqt1/assignment_loss‚2.843800164759159 /loss/FRt2/detection_loss‚1.8134871406946331 .loss/FRt3/detection_loss‚2.730346308648586 0loss/SRqqt1/detection_loss‚3.544166475534439 1loss/SRqqt2/detection_loss‚1.0900871266224763 0loss/FBt2/assignment_loss‚0.3270688993637691 1loss/SRqqt3/detection_loss‚1.4563581105563548 0loss/SRqqt4/detection_loss‚3.505323596622633 .loss/FBt2/detection_loss‚1.027033136660854 /loss/FRt4/detection_loss‚2.3565132804214954 /loss/FBt1/detection_loss‚2.6820683633839644 .loss/FBt3/detection_loss‚8.693558037281036 /loss/FRt2/assignment_loss‚3.161706004291773 _runtime‚ 837.692423707 % _timestamp‚1.7885112122221065e+09 0loss/FBt4/assignment_loss‚0.6233985513390508 epoch‚0 /loss/FRt3/assignment_loss‚4.241532619297504 0loss/SRqqt2/assignment_loss‚2.50051295175785 1loss/SRqqt3/assignment_loss‚2.260444980163079 2loss/SRqqt4/assignment_loss‚2.6910646220912104 1loss/FBt1/assignment_loss‚0.45916970369417454 $loss/total_loss‚69.8322101001346 /loss/FRt4/assignment_loss‚5.696795390999836 0loss/FBt3/assignment_loss‚0.8362333178520203 /loss/FRt1/detection_loss‚2.2829750080903373 _step‚59 .loss/FBt4/detection_loss‚7.1284706890583042|Ã&RÑÌ /loss/FRt2/detection_loss‚1.8134871406946331 .loss/FRt3/detection_loss‚2.730346308648586 0loss/SRqqt1/detection_loss‚3.544166475534439 1loss/SRqqt2/detection_loss‚1.0900871266224763 0loss/FBt2/assignment_loss‚0.3270688993637691 1loss/SRqqt3/detection_loss‚1.4563581105563548 0loss/SRqqt4/detection_loss‚3.505323596622633 .loss/FBt2/detection_loss‚1.027033136660854 /loss/FRt4/detection_loss‚2.3565132804214954 /loss/FBt1/detection_loss‚2.6820683633839644 .loss/FBt3/detection_loss‚8.693558037281036 /loss/FRt2/assignment_loss‚3.161706004291773 _runtime‚ 837.692423707 % _timestamp‚1.7885112122221065e+09 0loss/FBt4/assignment_loss‚0.6233985513390508 epoch‚0 /loss/FRt3/assignment_loss‚4.241532619297504 0loss/SRqqt2/assignment_loss‚2.50051295175785 1loss/SRqqt3/assignment_loss‚2.260444980163079 2loss/SRqqt4/assignment_loss‚2.6910646220912104 1loss/FBt1/assignment_loss‚0.45916970369417454 $loss/total_loss‚69.8322101001346 /loss/FRt4/assignment_loss‚5.696795390999836 0loss/FBt3/assignment_loss‚0.8362333178520203 /loss/FRt1/detection_loss‚2.2829750080903373 _step‚59 .loss/FBt4/detection_loss‚7.128470689058304 trainer/global_step‚749 /loss/FRt1/assignment_loss‚5.880095620950063 1loss/SRqqt1/assignment_loss‚2.843800164759159;ò=öå“Ò‚(:ˆ î‡êÔïÆ©ÿ gpu.0.gpu‚32.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.195816040039062* gpu.0.memoryAllocatedBytes‚ 14771093504 gpu.0.temp‚38.0 gpu.0.powerWatts‚176.738' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚44.1845 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0±GºûÓ‚(:† î‡êÔ«ªÕÿ disk.nvme0n1.out‚ 4076.15625 disk.sda2.in‚0* proc.memory.percent‚17.659974098205566 network.sent‚ 1.042175e+06) disk./.usagePercent‚75.76373143424114% disk./.usageGB‚2709.6780166625977 disk.sda.out‚94.69775390625 disk.sdb1.out‚0 disk.md0.in‚0 disk.md0.out‚ 91.37890625 disk.sdb2.in‚0$ memory_percent‚91.30864143371582# proc.memory.rssMB‚ 2893.41015625 disk.md0p1.out‚ 91.37890625 disk.sdb.out‚94.69775390625 disk.sdb2.out‚94.69775390625( proc.memory.availableMB‚ 1423.9921875 disk.sda1.out‚0 disk.nvme0n1.in‚ 0.02734375 disk.sdb.in‚0 disk.nvme1n1.out‚ 4187.90625 disk.sdb1.in‚0 disk.sda1.in‚0 disk.rbd2.in‚0 disk.sda.in‚0 disk.rbd0.in‚0 disk.nvme1n1.in‚ 0.05078125 proc.cpu.threads‚17 network.recv‚354187 disk.dm-0.in‚0.078125 disk.md0p1.in‚0 disk.sda2.out‚94.69775390625 disk.rbd0.out‚0 cpu‚25.069733661641777 disk.rbd2.out‚0 disk.dm-0.out‚ 8264.0625î™- ’Ô‚(:‡ ý‡êÔèñ€ gpu.0.gpu‚55.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.195816040039062* gpu.0.memoryAllocatedBytes‚ 14771093504 gpu.0.temp‚39.0 gpu.0.powerWatts‚173.676' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.419 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚00¾ÑFÕ‚(:‹ ý‡êÔìÓÝ€% disk./.usageGB‚2709.7794799804688 disk.nvme0n1.in‚ 0.02734375 cpu‚25.069049646900243 proc.cpu.threads‚17 disk.sda1.out‚0 disk.sda2.in‚0 disk.rbd0.out‚0 disk.nvme1n1.in‚ 0.05078125* proc.memory.percent‚17.659974098205566( disk./.usagePercent‚75.7665683910745" disk.nvme1n1.out‚ 4189.01171875 disk.rbd2.out‚0 disk.sdb2.in‚0 disk.sdb2.out‚96.16259765625 disk.nvme0n1.out‚ 4133.921875# proc.memory.rssMB‚ 2893.41015625 disk.rbd0.in‚0 disk.sdb1.out‚0 disk.md0.in‚0 disk.md0.out‚ 92.7890625 disk.md0p1.out‚ 92.7890625 disk.sda.out‚96.16259765625 disk.sda2.out‚96.16259765625 disk.sda1.in‚0 disk.dm-0.in‚0.078125 disk.dm-0.out‚ 8322.93359375 disk.md0p1.in‚0 disk.sda.in‚0$ memory_percent‚91.30616188049316( proc.memory.availableMB‚ 1424.3984375 network.sent‚ 1.089574e+06 disk.sdb1.in‚0 disk.rbd2.in‚0 disk.sdb.in‚0 disk.sdb.out‚96.16259765625 network.recv‚357684Gô0”Ö‚(:‰ ŒˆêÔ‚¾äý gpu.0.gpu‚56.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.195816040039062* gpu.0.memoryAllocatedBytes‚ 14771093504 gpu.0.temp‚39.0 gpu.0.powerWatts‚162.481' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚40.62025 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0<ÎŒI ׂ(:€ ŒˆêÔÓêÛþ disk.dm-0.in‚0.078125 disk.nvme0n1.out‚ 4204.859375 disk.sda2.out‚97.61962890625 disk.sdb1.out‚0* proc.memory.percent‚17.659997940063477 disk.nvme1n1.in‚ 0.05078125 disk.rbd2.out‚0 network.sent‚ 1.095581e+06 network.recv‚361965 disk.dm-0.out‚ 8394.265625 disk.sda.in‚0 disk.rbd0.in‚0 disk.md0.in‚0" proc.memory.rssMB‚ 2893.4140625 disk.sdb.in‚0 disk.sdb.out‚97.61962890625 proc.cpu.threads‚17 disk.sda.out‚97.61962890625 disk.sda1.in‚0 disk.md0p1.in‚0 disk.md0p1.out‚94.1875 disk.nvme0n1.in‚ 0.02734375 disk.sdb1.in‚0 disk.rbd2.in‚0 disk.sdb2.out‚97.61962890625 cpu‚25.067731954889076$ disk./.usageGB‚2709.917957305908 disk.sda2.in‚0 disk.sdb2.in‚0) proc.memory.availableMB‚ 1424.62890625) disk./.usagePercent‚75.77044027505107 disk.sda1.out‚0 disk.rbd0.out‚0 disk.nvme1n1.out‚ 4189.40625 disk.md0.out‚94.1875$ memory_percent‚91.30475521087646{Ü+Í“Ø‚(:ˆ ›ˆêÔ®¿‹þ gpu.0.gpu‚14.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.195816040039062* gpu.0.memoryAllocatedBytes‚ 14771093504 gpu.0.temp‚38.0 gpu.0.powerWatts‚169.942' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.4855 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0@çò(Ù‚(:† ›ˆêÔø»¡ÿ disk.dm-0.out‚ 8667.17578125 disk.sdb.out‚99.95751953125# proc.memory.rssMB‚ 2893.41796875 disk.sda.out‚99.95751953125 network.sent‚ 1.101298e+06) disk./.usagePercent‚75.77740776765138 disk.rbd2.out‚0 disk.dm-0.in‚0.078125 disk.md0p1.in‚0 disk.sdb.in‚0 network.recv‚365929! disk.nvme0n1.out‚ 4477.6328125 disk.rbd0.in‚0$ memory_percent‚91.30630493164062$ disk./.usageGB‚2710.167148590088 disk.nvme0n1.in‚ 0.02734375 disk.sda.in‚0 disk.rbd0.out‚0 disk.sdb1.in‚0$ proc.memory.availableMB‚1424.375 proc.cpu.threads‚17 disk.sdb2.in‚0 disk.sdb2.out‚99.95751953125 disk.md0p1.out‚ 96.453125 disk.md0.out‚ 96.453125* proc.memory.percent‚17.660021781921387 disk.sda2.in‚0 disk.nvme1n1.in‚ 0.05078125 disk.sdb1.out‚0 disk.rbd2.in‚0 disk.sda1.in‚0 disk.md0.in‚0 cpu‚25.067408573452074 disk.sda1.out‚0 disk.sda2.out‚99.95751953125" disk.nvme1n1.out‚ 4189.542968753D§ 0Ú+ _wandbruntime‚888  _runtime‚888Š¡zÞÃÛ½ trainer/global_step‚799 % _timestamp‚1.7885112633928652e+09 _runtime‚ 888.86331448 _step‚60 & lr-AdamW/pg1‚0.0005668343918312353 & lr-AdamW/pg2‚0.0005668343918312353›†ÇÜÁ _step‚60 & lr-AdamW/pg1‚0.0005668343918312353 & lr-AdamW/pg2‚0.0005668343918312353 trainer/global_step‚799 % _timestamp‚1.7885112633928652e+09 _runtime‚ 888.86331448DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24590 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1176280064 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9906853 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30433 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚15929054 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚14500121 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚64619755762688 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚15929056 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30188 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚72404363891712 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4593268 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 251469824 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14218690560 trainer/global_step‚799 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚15929054 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚63729640282624 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚563 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚19741467 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚64619755762688 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚72404112421888 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914443232 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14218690560 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚19716877 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 246679040 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚63293337921112 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1180696576 _runtime‚ 888.864639496 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 891014120960 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1189 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚63292423477880 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚563 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚63729658285568 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1180696576 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚35645931 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 886788089960 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚71475675854848 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 227529728 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510207117 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 890115480064 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚64620672406528 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 916643840 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 898640896 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚35670523 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚14500758 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9906855 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14218690560 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚64620672406528 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚63729658285568 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13954824192 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚71475699794944 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚15929056 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 890115480064 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚19716877 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚563 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24592 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 928664096768 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 928436567040 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30188 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1164 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1170 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚63729640282624 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30433 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24590 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚635 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 887685493832 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897403872 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚64619755762688 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚72404112421888 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914443232 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14218690560 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚19716877 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 246679040 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚63293337921112 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1180696576 _runtime‚ 888.864639496 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 891014120960 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1189 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚63292423477880 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚563 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚63729658285568 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1180696576 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚35645931 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 886788089960 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚71475675854848 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 227529728 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510207117 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 890115480064 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚64620672406528 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 916643840 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 898640896 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚35670523 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚14500758 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9906855 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14218690560 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚64620672406528 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚63729658285568 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13954824192 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚71475699794944 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚15929056 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 890115480064 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚19716877 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚563 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24592 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 928664096768 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 928436567040 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30188 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1164 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1170 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚63729640282624 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30433 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24590 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚635 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 887685493832 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897403872 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24590 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1176280064 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9906853 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30433 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚15929054 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚14500121 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚64619755762688 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚15929056 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30188 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚72404363891712 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4593268 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 251469824 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14218690560=¢Nôæ0à+ _wandbruntime‚889  _runtime‚889ž þkØ&áÒM NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚15948952 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1174999693 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚563 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚64701415518720 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 898654208 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚14518854 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚643 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897404080 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1180696576 $ _timestamp‚1.788511264360628e+09 _step‚62 ODeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 916657152 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚35715038 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚71565019015680 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 _runtime‚ 889.831414691 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚35690420 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1176280064 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 916657152 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚15948954 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510207117 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1170 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24618 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚14518211 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚641 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 888794155358 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13954824192 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚35690420 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 929819088896 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9919229 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14218690560 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚71564995075584 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚63372426676766 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1170 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚563 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚19766084 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚63371512233326 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚19766084 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚19741468 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 898654208 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚63809270578176 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 892126937600 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30433 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1170 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30433 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14218690560 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 891228283392 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚563 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚15948952 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚63809270578176 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚72494580356608 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚63809288581120 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1164 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚62483615482048 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 trainer/global_step‚799 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24616 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚15948954 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚64700498861568 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13954824192 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 929585281024 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1170 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 887896751278 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1189 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚72494838104576 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 233807872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 892126937600 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1176280064 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9919231 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4598982 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1180696576 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚19741468 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚35715038 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13954824192 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 929585281024 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1170 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 887896751278 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1189 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚72494838104576 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 233807872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 892126937600 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1176280064 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9919231 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4598982 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1180696576 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚19741468 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚35715038 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 916657152 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚35715038 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚71565019015680 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 _runtime‚ 889.831414691 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚35690420 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1176280064 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 916657152 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚15948954 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510207117 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1170 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24618 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚14518211 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚641 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 888794155358 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13954824192 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚35690420 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 929819088896 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9919229 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14218690560 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚71564995075584 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚63372426676766 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1170 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚563 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚19766084 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚63371512233326 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚19766084 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚19741468 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 898654208 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚63809270578176 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 892126937600 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30433 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1170 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0> S]0ã+ _wandbruntime‚889  _runtime‚889A7:¾Çä¹ .loss/FBt3/detection_loss‚1.661714792251587 $ _timestamp‚1.788511264362306e+09 0loss/FRt3/assignment_loss‚3.7217148828506468 .loss/FBt1/detection_loss‚2.797047535055562 /loss/zµxFBt2/detection_loss‚0.9800059606528116 /loss/FRt4/assignment_loss‚5.609012971605574 1loss/SRqqt2/assignment_loss‚2.329168397678088 0loss/SRqqt1/detection_loss‚2.853176755209764 1loss/SRqqt2/detection_loss‚1.2920337028495252 1loss/SRqqt3/detection_loss‚1.5119807873644047 epoch‚0 _runtime‚ 889.832419058 2loss/SRqqt4/assignment_loss‚2.4744250631332396 trainer/global_step‚799 /loss/FBt1/assignment_loss‚0.710913120718607 .loss/FRt2/detection_loss‚2.050689416773179 -loss/FBt4/detection_loss‚3.36514521141847 /loss/FRt1/detection_loss‚3.1578182528416314 /loss/FRt1/assignment_loss‚5.532020449638367 1loss/SRqqt1/assignment_loss‚2.340766289193804 1loss/SRqqt3/assignment_loss‚2.027252858254447 1loss/FBt2/assignment_loss‚0.32726559676683564 /loss/FRt3/detection_loss‚2.2942403244972227 /loss/FRt4/detection_loss‚2.5257191798516683 1loss/SRqqt4/detection_loss‚3.0017430233955382 /loss/FRt2/assignment_loss‚2.619303228808384 0loss/FBt4/assignment_loss‚0.6891452560271318 &loss/total_loss‚55.872303056836486 _step‚63 loss/FBt3/assignment_loss‚0w ŽCå½ 1loss/SRqqt4/detection_loss‚3.0017430233955382 /loss/FRt2/assignment_loss‚2.619303228808384 0loss/FBt4/assignment_loss‚0.6891452560271318 &loss/total_loss‚55.872303056836486 _step‚63 loss/FBt3/assignment_loss‚0 .loss/FBt3/detection_loss‚1.661714792251587 $ _timestamp‚1.788511264362306e+09 0loss/FRt3/assignment_loss‚3.7217148828506468 .loss/FBt1/detection_loss‚2.797047535055562 /loss/FBt2/detection_loss‚0.9800059606528116 /loss/FRt4/assignment_loss‚5.609012971605574 1loss/SRqqt2/assignment_loss‚2.329168397678088 0loss/SRqqt1/detection_loss‚2.853176755209764 1loss/SRqqt2/detection_loss‚1.2920337028495252 1loss/SRqqt3/detection_loss‚1.5119807873644047 epoch‚0 _runtime‚ 889.832419058 2loss/SRqqt4/assignment_loss‚2.4744250631332396 trainer/global_step‚799 /loss/FBt1/assignment_loss‚0.710913120718607 .loss/FRt2/detection_loss‚2.050689416773179 -loss/FBt4/detection_loss‚3.36514521141847 /loss/FRt1/detection_loss‚3.1578182528416314 /loss/FRt1/assignment_loss‚5.532020449638367 1loss/SRqqt1/assignment_loss‚2.340766289193804 1loss/SRqqt3/assignment_loss‚2.027252858254447 1loss/FBt2/assignment_loss‚0.32726559676683564 /loss/FRt3/detection_loss‚2.2942403244972227 /loss/FRt4/detection_loss‚2.5257191798516683?§™£t”æ‚(:‰ ªˆêÔŽý˜ÿ gpu.0.gpu‚40.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.195816040039062* gpu.0.memoryAllocatedBytes‚ 14771093504 gpu.0.temp‚37.0 gpu.0.powerWatts‚123.095' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚30.77375 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0w…ëç‚(:† ªˆêÔéŤÿ disk.sda.in‚0 disk.sda2.in‚0 disk.md0.in‚0 disk.sdb1.in‚0 disk.sda1.out‚0 disk.md0p1.in‚0! proc.memory.rssMB‚ 2893.421875 network.recv‚370683 disk.nvme1n1.in‚ 0.05078125 disk.sdb1.out‚0' proc.memory.availableMB‚ 1424.734375 network.sent‚ 1.107521e+06$ disk./.usageGB‚2710.171230316162 disk.sda.out‚101.72900390625! disk.sdb2.out‚101.72900390625$ memory_percent‚91.30411148071289* proc.memory.percent‚17.660045623779297 cpu‚25.066314554841014 disk.sdb.in‚0! disk.sda2.out‚101.72900390625 disk.md0.out‚98.15625 disk.rbd2.out‚0 disk.md0p1.out‚98.15625) disk./.usagePercent‚75.77752189442076 disk.sdb.out‚101.72900390625 disk.sdb2.in‚0 disk.nvme0n1.in‚ 0.02734375 proc.cpu.threads‚17 disk.nvme1n1.out‚ 4189.9375 disk.rbd2.in‚0 disk.sda1.in‚0 disk.dm-0.in‚0.078125 disk.dm-0.out‚ 8719.95703125 disk.rbd0.in‚0 disk.rbd0.out‚0" disk.nvme0n1.out‚ 4530.01953125ÇT’è‚(:‡ ¹ˆêÔج™þ gpu.0.gpu‚45.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.195816040039062* gpu.0.memoryAllocatedBytes‚ 14771093504 gpu.0.temp‚38.0 gpu.0.powerWatts‚173.676' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.419 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0{"Ýé‚(:ƒ ¹ˆêÔ«òÿ disk.sda.in‚0 disk.sda1.in‚0 disk.nvme0n1.out‚ 4597.46875 disk.rbd0.out‚0 disk.sdb1.in‚0 disk.md0.out‚99.625! disk.sdb2.out‚103.26123046875 disk.dm-0.in‚0.078125 disk.md0p1.out‚99.625 network.recv‚374453 disk.sda.out‚103.26123046875! disk.sda2.out‚103.26123046875 disk.dm-0.out‚ 8787.5703125 disk.sdb.in‚0! proc.memory.rssMB‚ 2893.421875 disk.nvme1n1.in‚ 0.05078125 disk.sdb1.out‚0 disk.sdb.out‚103.26123046875$ disk./.usageGB‚2710.171230316162 disk.md0.in‚0 disk.rbd2.in‚0 disk.nvme0n1.in‚ 0.02734375$ memory_percent‚91.30547046661377 cpu‚25.066064101132046 disk.sda2.in‚0 disk.md0p1.in‚0* proc.memory.percent‚17.660045623779297 network.sent‚ 1.155689e+06 disk.sdb2.in‚0 disk.sda1.out‚0) proc.memory.availableMB‚ 1424.51171875 proc.cpu.threads‚17) disk./.usagePercent‚75.77752189442076 disk.rbd0.in‚0! disk.nvme1n1.out‚ 4190.1015625 disk.rbd2.out‚0Ác[Ç”ê‚(:‰ ȈêÔ¹‚…þ gpu.0.gpu‚12.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.195816040039062* gpu.0.memoryAllocatedBytes‚ 14771093504 gpu.0.temp‚39.0 gpu.0.powerWatts‚176.461' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚44.11525 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0v{Öãë‚(:‘ ȈêÔƒÔÿþ) disk./.usagePercent‚75.77926771401053$ disk./.usageGB‚2710.233669281006 disk.sdb1.in‚0! disk.nvme0n1.out‚ 4699.5859375 disk.sda.in‚0 disk.sda2.in‚0 disk.sdb.out‚105.28369140625 disk.rbd0.out‚0" disk.nvme1n1.out‚ 4190.34765625 disk.sdb2.in‚0! disk.sdb2.out‚105.28369140625 disk.nvme0n1.in‚ 0.02734375$ memory_percent‚91.30465984344482 network.recv‚378935 disk.sdb1.out‚0 disk.rbd2.in‚0) proc.memory.availableMB‚ 1424.64453125! proc.memory.rssMB‚ 2893.421875* proc.memory.percent‚17.660045623779297 proc.cpu.threads‚17 disk.rbd0.in‚0 disk.nvme1n1.in‚ 0.05078125 disk.dm-0.out‚ 8889.93359375 cpu‚25.065294705144577 disk.sda1.in‚0 disk.sda.out‚105.28369140625 network.sent‚ 1.161637e+06 disk.md0.in‚0 disk.md0.out‚ 101.5859375 disk.sda1.out‚0 disk.md0p1.out‚ 101.5859375 disk.sdb.in‚0 disk.dm-0.in‚0.078125 disk.md0p1.in‚0 disk.rbd2.out‚0! disk.sda2.out‚105.28369140625êŒO#0ì+ _wandbruntime‚940  _runtime‚940fÉÀÅÃí½ _step‚64 & lr-AdamW/pg1‚0.0006023058806817505 & lr-AdamW/pg2‚0.0006023058806817505 trainer/global_step‚849 $ _timestamp‚1.788511314822519e+09 _runtime‚ 940.292949349Ðe›_ÇîÁ trainer/global_step‚849 $ _timestamp‚1.788511314822519e+09 _runtime‚ 940.292949349 _step‚64 & lr-AdamW/pg1‚0.0006023058806817505 & lr-AdamW/pg2‚0.0006023058806817505@E‘Ę0ï+ _wandbruntime‚940  _runtime‚940Ìá^Ð'ðÊO QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 916643840 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚66304657133680 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1176280064 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1164 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚20974369 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚15406969 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10525653 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚20974369 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚16923956 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4881932 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1170 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24592 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897403872 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚10525655 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1176280064 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 898640896 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚16923954 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚616 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚563 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚16923956 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24590 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚76928108638720 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 398533632 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914443232 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚37873733 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30188 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13954824192 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24592 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚67711155060224 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 898640896 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚37898325 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 986703471104 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 986471747072 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚67247789284620 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30188 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 945770060288 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚67246874841388 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _step‚65 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚67711155060224 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚75941660831744 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1174999693 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1180696576 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚20949779 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚563 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚68657841764352 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24590 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 916643840 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 945770060288 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1180696576 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1170 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚20949779 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚76928364302848 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚75941636891648 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚66304640094320 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 231724032 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510207117 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚68656925120512 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30433 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4881316 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚67711173063168 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 946668701184 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 943132150940 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1170 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14218690560 trainer/global_step‚849 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚68656925120512 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30433 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 255664128 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚618 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13954824192 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 942234747068 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14218690560 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 $ _timestamp‚1.788511314823623e+09 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚68657841764352 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 946668701184 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚15407587 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 246679040 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1180696576 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚67711173063168 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1170 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚68656925120512 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30433 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4881316 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚67711173063168 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 946668701184 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 943132150940 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1170 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14218690560 trainer/global_step‚849 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚68656925120512 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30433 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 255664128 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚618 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13954824192 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 942234747068 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14218690560 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 $ _timestamp‚1.788511314823623e+09 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚68657841764352 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 946668701184 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚15407587 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 246679040 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1180696576 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚67711173063168 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1170 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 398533632 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914443232 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚37873733 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30188 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13954824192 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24592 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚67711155060224 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 898640896 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚37898325 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 986703471104 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 986471747072 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚67247789284620 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30188 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 945770060288 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚67246874841388 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _step‚65 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚67711155060224 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚75941660831744 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0A„ÿÂó0ò+ _wandbruntime‚941  _runtime‚941 >m"óÕM J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚16943852 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30188 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1180696576 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13954824192 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 946882863616 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 253553664 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚20974370 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚37942840 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1170 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚20974370 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚15425758 trainer/global_step‚849 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚20998986 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚67790785355776 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚76030980052480 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1176280064 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚67790785355776 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13954824192 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1176280064 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 246679040 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1180696576 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚37918222 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510207117 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚66382637227808 _runtime‚ 941.407763346 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚77018583330816 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1170 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚563 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24618 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10538029 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1164 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24616 ODeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 916657152 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 947781517824 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 946882863616 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚66382620188448 _step‚66 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚68738584876544 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚68738584876544 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 947781517824 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚37918222 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1189 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4887727 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚15425132 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚563vx6Lß&ôÙM J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚20974370 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚37942840 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1170 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚20974370 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚15425758 trainer/global_step‚849 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚20998986 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚67790785355776 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚76030980052480 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1176280064 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚67790785355776 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13954824192 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1176280064 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 246679040 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1180696576 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚37918222 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510207117 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚66382637227808 _runtime‚ 941.407763346 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚77018583330816 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1170 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚563 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24618 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10538029 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1164 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24616 ODeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 916657152 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 947781517824 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 946882863616 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚66382620188448 _step‚66 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚68738584876544 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚68738584876544 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 947781517824 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚37918222 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1189 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4887727 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚15425132 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚563 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚16943852 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30188 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1180696576 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13954824192 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 946882863616 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 253553664 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0BþÉÜ60õ+ _wandbruntime‚941  _runtime‚941ô±^PöÊ trainer/global_step‚849 2loss/SRqqt3/assignment_loss‚2.3205351531505585 2loss/SRqqt4/assignment_loss‚2.5335352569818497 1loss/SRqqt2/detection_loss‚1.3401095399899141 1loss/SRqqt4/detection_loss‚2.3747901434407517 %loss/total_loss‚60.08607297279178 _runtime‚ 941.409049492 2loss/SRqqt1/assignment_loss‚2.5873903243438057 /loss/FBt2/detection_loss‚0.6889648489697382 .loss/FBt4/detection_loss‚2.774942684918642 2loss/SRqqt2/assignment_loss‚1.9933105261198112 1loss/FBt2/assignment_loss‚0.33355739958202313 0loss/FBt3/assignment_loss‚0.6696814894676208 /loss/FBt4/assignment_loss‚0.790837462292977 0loss/SRqqt3/detection_loss‚2.191634544481834 % _timestamp‚1.7885113159387665e+09 /loss/FRt1/assignment_loss‚4.894410231174567 /loss/FRt2/assignment_loss‚2.521300347951742 _step‚67 /loss/FRt3/assignment_loss‚4.133810937404633 /loss/FRt1/detection_loss‚2.4664523601531982 .loss/FRt2/detection_loss‚2.383897457367335 /loss/FRt4/detection_loss‚2.7730536669492722 0loss/SRqqt1/detection_loss‚2.828425343917764 .loss/FBt3/detection_loss‚5.995790481567383 .loss/FRt4/assignment_loss‚5.42485466003418 0loss/FBt1/assignment_loss‚0.7384553809872159 /loss/FRt3/detection_loss‚2.3366385276118913 -loss/FBt1/detection_loss‚2.98969420393308 epoch‚0±Üi7T÷Î 0loss/FBt3/assignment_loss‚0.6696814894676208 /loss/FBt4/assignment_loss‚0.790837462292977 0loss/SRqqt3/detection_loss‚2.191634544481834 % _timestamp‚1.7885113159387665e+09 /loss/FRt1/assignment_loss‚4.894410231174567 /loss/FRt2/assignment_loss‚2.521300347951742 _step‚67 /loss/FRt3/assignment_loss‚4.133810937404633 /loss/FRt1/detection_loss‚2.4664523601531982 .loss/FRt2/detection_loss‚2.383897457367335 /loss/FRt4/detection_loss‚2.7730536669492722 0loss/SRqqt1/detection_loss‚2.828425343917764 .loss/FBt3/detection_loss‚5.995790481567383 .loss/FRt4/assignment_loss‚5.42485466003418 0loss/FBt1/assignment_loss‚0.7384553809872159 /loss/FRt3/detection_loss‚2.3366385276118913 -loss/FBt1/detection_loss‚2.98969420393308 epoch‚0 trainer/global_step‚849 2loss/SRqqt3/assignment_loss‚2.3205351531505585 2loss/SRqqt4/assignment_loss‚2.5335352569818497 1loss/SRqqt2/detection_loss‚1.3401095399899141 1loss/SRqqt4/detection_loss‚2.3747901434407517 %loss/total_loss‚60.08607297279178 _runtime‚ 941.409049492 2loss/SRqqt1/assignment_loss‚2.5873903243438057 /loss/FBt2/detection_loss‚0.6889648489697382 .loss/FBt4/detection_loss‚2.774942684918642 2loss/SRqqt2/assignment_loss‚1.9933105261198112 1loss/FBt2/assignment_loss‚0.33355739958202313CÈ®Œ ø‚(:’ ׈êÔ…êÿ gpu.0.gpu‚7.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.195816040039062* gpu.0.memoryAllocatedBytes‚ 14771093504 gpu.0.temp‚37.0 gpu.0.powerWatts‚115.634' gpu.0.enforcedPowerLimitWatts‚400.0) gpu.0.powerPercent‚28.908499999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0õï¸ù‚(:‡ ׈êÔÅ¡€€ disk.sda1.in‚0) proc.memory.availableMB‚ 1424.21484375 disk.rbd2.in‚0 disk.sda1.out‚0) disk./.usagePercent‚75.76983231001797 disk.rbd0.in‚0 disk.md0.in‚0 disk.dm-0.out‚ 9011.63671875 disk.nvme0n1.in‚ 0.02734375 disk.sdb.out‚107.02197265625 disk.rbd0.out‚0 disk.nvme1n1.in‚ 0.05078125! disk.sdb2.out‚107.02197265625! proc.memory.rssMB‚ 2893.421875$ disk./.usageGB‚2709.896213531494 disk.sda.out‚107.02197265625 disk.dm-0.in‚0.078125* proc.memory.percent‚17.660045623779297 network.sent‚ 1.180254e+06 disk.sdb.in‚0! disk.sda2.out‚107.02197265625 disk.sdb1.out‚0 disk.rbd2.out‚0 cpu‚25.064746157179993 proc.cpu.threads‚17 disk.sdb1.in‚0 disk.md0.out‚103.25 disk.md0p1.in‚0 disk.md0p1.out‚103.25$ memory_percent‚91.30728244781494 network.recv‚382904! disk.nvme1n1.out‚ 4190.7578125" disk.nvme0n1.out‚ 4820.87890625 disk.sda.in‚0 disk.sda2.in‚0 disk.sdb2.in‚0»‚§“ú‚(:ˆ æˆêԻȂ€ gpu.0.gpu‚2.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.195816040039062* gpu.0.memoryAllocatedBytes‚ 14771093504 gpu.0.temp‚37.0 gpu.0.powerWatts‚159.419' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚39.85475 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 vû‚(: æˆêÔÝÛ¡ disk.dm-0.out‚ 9160.73046875 disk.sdb.in‚0 disk.rbd0.in‚0 disk.rbd2.in‚0 disk.rbd2.out‚0# proc.memory.rssMB‚ 2893.42578125 cpu‚25.06445572437834) disk./.usagePercent‚75.76980276505057 disk.nvme0n1.in‚ 0.02734375! disk.sda2.out‚108.55615234375 disk.sda1.out‚0$ memory_percent‚91.30630493164062 disk.sda2.in‚0 disk.nvme1n1.in‚ 0.05078125 disk.sdb1.out‚0! disk.sdb2.out‚108.55615234375* proc.memory.percent‚17.660069465637207 disk.md0.out‚ 104.72265625$ proc.memory.availableMB‚1424.375% disk./.usageGB‚2709.8951568603516" disk.nvme1n1.out‚ 4191.06640625 network.recv‚388123! disk.nvme0n1.out‚ 4969.6640625 disk.md0.in‚0 disk.sdb2.in‚0 network.sent‚ 1.218842e+06 disk.md0p1.in‚0 disk.sdb.out‚108.55615234375 disk.sda.out‚108.55615234375 disk.sdb1.in‚0 disk.dm-0.in‚0.078125 disk.md0p1.out‚ 104.72265625 disk.sda.in‚0 disk.rbd0.out‚0 disk.sda1.in‚0 proc.cpu.threads‚17nÍ+”ü‚(:‰ õˆêÔáðÒý gpu.0.gpu‚18.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.195816040039062* gpu.0.memoryAllocatedBytes‚ 14771093504 gpu.0.temp‚38.0 gpu.0.powerWatts‚174.903' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.72575 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‰TýTý‚(:‰ õˆêÔþ÷»ÿ disk.dm-0.out‚ 9261.484375 proc.cpu.threads‚17 network.recv‚391685) disk./.usagePercent‚75.77648547403012 disk.sda.out‚110.15576171875 disk.sda2.in‚0 disk.sdb1.out‚0 disk.nvme0n1.in‚ 0.02734375$ disk./.usageGB‚2710.134162902832 disk.nvme1n1.in‚ 0.05078125 disk.rbd2.out‚0( proc.memory.availableMB‚ 1424.2578125 network.sent‚ 1.222039e+06 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda1.out‚0 disk.sdb.in‚0# proc.memory.rssMB‚ 2893.42578125 disk.rbd2.in‚0! disk.sdb2.out‚110.15576171875 disk.sda1.in‚0 disk.md0p1.in‚0 disk.nvme0n1.out‚5070.25 disk.sdb.out‚110.15576171875$ memory_percent‚91.30702018737793! disk.sda2.out‚110.15576171875 disk.md0.out‚ 106.2578125 disk.dm-0.in‚0.078125 disk.md0p1.out‚ 106.2578125* proc.memory.percent‚17.660069465637207 cpu‚25.063832188618022 disk.sda.in‚0 disk.nvme1n1.out‚ 4191.234375 disk.sdb1.in‚0 disk.md0.in‚0 disk.sdb2.in‚0^€Ø­”þ‚(:‰ „‰êÔç«»ÿ gpu.0.gpu‚16.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.195816040039062* gpu.0.memoryAllocatedBytes‚ 14771093504 gpu.0.temp‚37.0 gpu.0.powerWatts‚117.193' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚29.29825 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0\šrHÿ‚(:½ „‰êÔÐÓ€ disk.sda.out‚112.06591796875" disk.nvme1n1.out‚ 4191.56640625 disk.rbd2.in‚0" disk.nvme0n1.out‚ 5347.55859375 disk.sda2.in‚ 0.0390625 disk.sdb1.in‚0 disk.sdb1.out‚0 cpu‚25.063348001724886 proc.cpu.threads‚17 network.sent‚ 1.227959e+06 disk.md0p1.out‚ 108.10546875 disk.nvme0n1.in‚ 0.02734375$ memory_percent‚91.30880832672119) disk./.usagePercent‚75.77757127824712 disk.sda.in‚ 0.0390625! disk.sdb2.out‚112.06591796875) proc.memory.availableMB‚ 1423.96484375 disk.sda1.out‚0 network.recv‚396143$ disk./.usageGB‚2710.172996520996! disk.sda2.out‚112.06591796875 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.rbd2.out‚0 disk.sdb2.in‚0.03125 disk.dm-0.in‚0.078125 disk.md0p1.in‚ 0.0703125 disk.nvme1n1.in‚ 0.05078125 disk.md0.in‚ 0.0703125 disk.sda1.in‚0# proc.memory.rssMB‚ 2893.42578125* proc.memory.percent‚17.660069465637207 disk.dm-0.out‚9539.125 disk.sdb.out‚112.06591796875 disk.md0.out‚ 108.10546875 disk.sdb.in‚0.03125pûAÎ0€+ _wandbruntime‚997  _runtime‚997 -p®Á» % lr-AdamW/pg1‚0.000637777369532266 % lr-AdamW/pg2‚0.000637777369532266 trainer/global_step‚899 $ _timestamp‚1.788511372121555e+09 _runtime‚ 997.591982142 _step‚68âØ Å‚¿ % lr-AdamW/pg1‚0.000637777369532266 % lr-AdamW/pg2‚0.000637777369532266 trainer/global_step‚899 $ _timestamp‚1.788511372121555e+09 _runtime‚ 997.591982142 _step‚68DßæÊu0ƒ+ _wandbruntime‚997  _runtime‚997P·¢O!„ÑO QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚633 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1044764486656 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1170 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚72694094478336 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13954824192 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 997681404176 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1170 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚40101535 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚72694094478336 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510207117 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚17918854 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚22207271 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚71202240648128 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚22207271 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚72695011122176 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1002323281408 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14218690560 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1180696576 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚71692669837824 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚635 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚80407621868544 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1170 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚563 % _timestamp‚1.7885113721225777e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚71692669837824 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚72695011122176 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚17918856 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1044530665472 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1180696576 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30188 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24592 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897403872 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1174999693 A7DevicàæC.ˆeStatsMonitor.on_train_batch_start/active.all.peak‚30433 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 246679040 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24590 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 916643840 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 916643840 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5171654 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚70203644800720 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24592 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1002323281408 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 898640896 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1176280064 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚16315474 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1001424640512 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1164 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 998578808048 trainer/global_step‚899 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚17918856 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1176280064KÖ„€Û'…ÕO WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1001424640512 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1164 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 998578808048 trainer/global_step‚899 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚17918856 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1176280064 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚633 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1044764486656 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1170 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚72694094478336 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13954824192 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 997681404176 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1170 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚40101535 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚72694094478336 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510207117 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚17918854 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚22207271 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚71202240648128 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚22207271 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚72695011122176 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1002323281408 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14218690560 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1180696576 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚71692669837824 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚635 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚80407621868544 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1170 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚563 % _timestamp‚1.7885113721225777e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚71692669837824 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚72695011122176 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚17918856 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1044530665472 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1180696576 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30188 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24592 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897403872 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1174999693 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30433 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 246679040 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24590 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 916643840 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 916643840 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5171654 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚70203644800720 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24592 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1002323281408 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 898640896 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1176280064 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚16315474E0M±0†+ _wandbruntime‚998  _runtime‚998˜üÓà&‡ÚM YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚70281624894848 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1180696576 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24616 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30433 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 _runtime‚ 998.867372747 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14218690560 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24616 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14218690560 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚80496917149184 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚16334247 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1003436098048 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚40170642 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚563 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚40146024 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚22207272 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚22207272 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1170 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚72775754234368 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 $ _timestamp‚1.788511373395544e+09 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚71280414960342 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897404080 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1180696576 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1045920805888 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚70281641934208 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1174999693 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚22231888 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30188 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1164 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1180696576 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1170 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1176280064 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5176780 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚81542861895168 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚563 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚71281329403782 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚40170642 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 916657152 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚71772300133376 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 916657152 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24618 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚16333609 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5177416 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚636 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914443440 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚17938754 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚71772318136320 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚81542612535808 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ODeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510207117 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1170 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚638 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 225419264 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13954824192 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 398533632 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚80496941089280 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14218690560 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚72775754234368 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚11156831 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 249359360 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1045695386624 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 999687469574 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚71772318136320 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚71772300133376 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30433 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0÷M.ä&ˆÞM UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚71772318136320 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚71772300133376 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30433 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚70281624894848 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1180696576 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24616 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30433 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 _runtime‚ 998.867372747 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14218690560 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24616 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14218690560 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚80496917149184 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚16334247 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1003436098048 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚40170642 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚563 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚40146024 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚22207272 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚22207272 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1170 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚72775754234368 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 $ _timestamp‚1.788511373395544e+09 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚71280414960342 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897404080 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1180696576 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1045920805888 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚70281641934208 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1174999693 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚22231888 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30188 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1164 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1180696576 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1170 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1176280064 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5176780 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚81542861895168 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚563 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚71281329403782 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚40170642 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 916657152 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚71772300133376 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 916657152 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24618 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚16333609 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5177416 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚636 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914443440 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚17938754 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚71772318136320 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚81542612535808 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ODeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510207117 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1170 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚638 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 225419264 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13954824192 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 398533632 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚80496941089280 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14218690560 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚72775754234368 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚11156831 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 249359360 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1045695386624 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 999687469574 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 FDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30188 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13954824192 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5460998 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 398533632 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚76731263836160 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚18913756 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1102721530880 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1174999693 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚76731263836160 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 223335424 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚563 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚18913754 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13954824192 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚23440173 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚11763255 trainer/global_step‚949 _step‚73 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1180696576 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24592 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚42353929 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚17223645 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24592 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚74102666546480 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚18913756 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚23440173 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1057977861632 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1189 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚563 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1057079220736 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1176280064 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚74102649507120 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1170 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 916643840 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30188 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5460392 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 247275520 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14218690560 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 916643840 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30433 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1170 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚23415583 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1057079220736 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510207117 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 246679040 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1170 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚563 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914443232 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14218690560cÇÂÝ'™×O $ _timestamp‚1.788511431864745e+09 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚75674184615424 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚11763253 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1180696576 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚76731263836160 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚18913756 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1102721530880 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1174999693 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚76731263836160 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 223335424 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚563 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚18913754 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13954824192 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚23440173 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚11763255 trainer/global_step‚949 _step‚73 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1180696576 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24592 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚42353929 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚17223645 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24592 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚74102666546480 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚18913756 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚23440173 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1057977861632 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1189 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚563 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1057079220736 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1176280064 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚74102649507120 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1170 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 916643840 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30188 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5460392 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 247275520 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14218690560 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 916643840 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30433 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1170 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚23415583 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1057079220736 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510207117 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 246679040 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1170 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚563 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914443232 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14218690560 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1164 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚84873558965248 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1102498195456 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30433 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚75674202618368 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚75674184615424 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 898640896 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897403872 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1053128061284 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30188 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13954824192 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5460998 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 398533632 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510207117 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30188 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14218690560 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24616 ODeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13954824192 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1058192024064 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚23440174 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 261942272 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 246679040 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚23464790 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1059090678272 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30433 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1103873571840 EòÖFü=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚42398444 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚75753814910976 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 898654208 _runtime‚1058.545142779 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚18933654 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚563 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚74180646640608 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1174999693 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 $ _timestamp‚1.788511433073021e+09 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚42373826 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚74180629601248 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚42398444 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚17241749 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1170 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚42373826 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚86066775697920 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _step‚74 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚23440174 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 916657152 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 898654208 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚563 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1176280064 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1103635569664 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1180696576 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13954824192 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚76812923592192 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 916657152 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1054236722602 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1164 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚75234866323850 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1170 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚563 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24618 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 trainer/global_step‚949¬ØÏ;ç&œáM D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510207117 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30188 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14218690560 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24616 ODeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13954824192 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1058192024064 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚23440174 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 261942272 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 246679040 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚23464790 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1059090678272 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30433 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1103873571840 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚42398444 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚75753814910976 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 898654208 _runtime‚1058.545142779 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚18933654 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚563 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚74180646640608 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1174999693 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 $ _timestamp‚1.788511433073021e+09 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚42373826 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚74180629601248 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚42398444 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚17241749 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1170 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚42373826 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚86066775697920 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _step‚74 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚23440174 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 916657152 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 898654208 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚563 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1176280064 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1103635569664 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1180696576 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13954824192 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚76812923592192 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 916657152 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1054236722602 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1164 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚75234866323850 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1170 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚563 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24618 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 trainer/global_step‚949 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 398533632 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1170 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14218690560 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1170J–´ÿ/2- _wandbruntime‚1058  _runtime‚1058͈rZTžÎ 2loss/SRqqt1/assignment_loss‚2.2155603091082265 2loss/SRqqt3/assignment_loss‚1.7336127338275102 1loss/FBt1/assignment_loss‚0.43640953030585067 /loss/FRt2/detection_loss‚2.1178722828626633 /loss/FBt1/detection_loss‚3.2470904263583096 epoch‚0 1loss/FBt2/assignment_loss‚0.23375134632890296 1loss/SRqqt2/detection_loss‚1.6050603528294647 /loss/FBt2/detection_loss‚0.8397517876971202 &loss/total_loss‚56.679333412830076 _step‚75 /loss/FRt4/assignment_loss‚5.151207759976387 0loss/SRqqt1/detection_loss‚2.522331799230268 _runtime‚ 1058.54763392 /loss/FRt1/detection_loss‚2.5339550359829053 /loss/FRt3/detection_loss‚2.2184207439422607 .loss/FRt4/detection_loss‚2.892268158495426 1loss/SRqqt3/detection_loss‚1.5845474149559584 .loss/FBt3/detection_loss‚4.989474952220917 $ _timestamp‚1.788511433077415e+09 1loss/SRqqt4/assignment_loss‚2.525551432600388 .loss/FBt4/detection_loss‚3.315576307475567 /loss/FRt2/assignment_loss‚2.191717140376568 2loss/SRqqt2/assignment_loss‚1.3456273498503786 trainer/global_step‚949 0loss/FBt3/assignment_loss‚0.5676169693470001 0loss/FBt4/assignment_loss‚0.6510116457935737 0loss/SRqqt4/detection_loss‚2.802301830970324 /loss/FRt1/assignment_loss‚5.035579790940156 /loss/FRt3/assignment_loss‚3.923036311353956êñ”{XŸÒ /loss/FRt3/assignment_loss‚3.923036311353956 2loss/SRqqt1/assignment_loss‚2.2155603091082265 2loss/SRqqt3/assignment_loss‚1.7336127338275102 1loss/FBt1/assignment_loss‚0.43640953030585067 /loss/FRt2/detection_loss‚2.1178722828626633 /loss/FBt1/detection_loss‚3.2470904263583096 epoch‚0 1loss/FBt2/assignment_loss‚0.23375134632890296 1loss/SRqqt2/detection_loss‚1.6050603528294647 /loss/FBt2/detection_loss‚0.8397517876971202 &loss/total_loss‚56.679333412830076 _step‚75 /loss/FRt4/assignment_loss‚5.151207759976387 0loss/SRqqt1/detection_loss‚2.522331799230268 _runtime‚ 1058.54763392 /loss/FRt1/detection_loss‚2.5339550359829053 /loss/FRt3/detection_loss‚2.2184207439422607 .loss/FRt4/detection_loss‚2.892268158495426 1loss/SRqqt3/detection_loss‚1.5845474149559584 .loss/FBt3/detection_loss‚4.989474952220917 $ _timestamp‚1.788511433077415e+09 1loss/SRqqt4/assignment_loss‚2.525551432600388 .loss/FBt4/detection_loss‚3.315576307475567 /loss/FRt2/assignment_loss‚2.191717140376568 2loss/SRqqt2/assignment_loss‚1.3456273498503786 trainer/global_step‚949 0loss/FBt3/assignment_loss‚0.5676169693470001 0loss/FBt4/assignment_loss‚0.6510116457935737 0loss/SRqqt4/detection_loss‚2.802301830970324 /loss/FRt1/assignment_loss‚5.035579790940156K,±)h“ ‚(:ˆ ωêԮ݀ gpu.0.gpu‚36.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.195816040039062* gpu.0.memoryAllocatedBytes‚ 14771093504 gpu.0.temp‚37.0 gpu.0.powerWatts‚129.614' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚32.4035 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0“OI¡‚(:¾ ωêÔüŸÐ disk.sda.in‚ 0.0390625 disk.sda.out‚120.4619140625 disk.sda1.in‚0 disk.sda1.out‚0 disk.md0p1.in‚ 0.0703125 disk.sdb.in‚0.03125 disk.sdb.out‚120.4619140625# proc.memory.rssMB‚ 2893.42578125" disk.nvme0n1.out‚ 6168.65234375$ disk./.usageGB‚2710.673152923584 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.rbd2.in‚0 disk.sdb2.out‚120.4619140625 proc.cpu.threads‚17( disk./.usagePercent‚75.7915558605952 disk.nvme1n1.in‚ 0.05078125 disk.md0.in‚ 0.0703125 cpu‚25.0600653884894 disk.md0p1.out‚116.1875 network.recv‚417811 disk.rbd0.in‚0 disk.dm-0.in‚0.078125$ memory_percent‚91.30890369415283) proc.memory.availableMB‚ 1423.94921875 network.sent‚ 1.300316e+06 disk.sda2.out‚120.4619140625 disk.nvme1n1.out‚4192.75 disk.md0.out‚116.1875 disk.dm-0.out‚10361.40234375 disk.nvme0n1.in‚ 0.02734375 disk.sda2.in‚ 0.0390625 disk.rbd0.out‚ 0.01171875 disk.rbd2.out‚0.078125 disk.sdb2.in‚0.03125* proc.memory.percent‚17.660069465637207ÂA3 ”¢‚(:‰ Þ‰êÔŸî“ÿ gpu.0.gpu‚40.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.195816040039062* gpu.0.memoryAllocatedBytes‚ 14771093504 gpu.0.temp‚38.0 gpu.0.powerWatts‚172.727' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.18175 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Á,*SI£‚(:¾ Þ‰êÔ½Èíÿ network.sent‚ 1.35189e+06 disk.rbd0.out‚ 0.07421875 disk.rbd2.out‚0.078125 disk.md0p1.in‚ 0.0703125 network.recv‚423152% disk./.usageGB‚2710.1178436279297 disk.sda2.out‚121.9365234375 disk.nvme1n1.in‚ 0.05078125 disk.md0.out‚ 117.59375 disk.sdb2.in‚0.03125 disk.nvme0n1.out‚6272.875 cpu‚25.059559926715643) disk./.usagePercent‚75.77602918027371 disk.sda.in‚ 0.0390625$ memory_percent‚91.30630493164062 disk.sdb2.out‚121.9365234375 disk.sdb.out‚121.9365234375 disk.sdb1.out‚0 disk.md0.in‚ 0.0703125 disk.sda1.out‚0* proc.memory.percent‚17.660093307495117 proc.cpu.threads‚17 disk.sda.out‚121.9365234375$ proc.memory.availableMB‚1424.375 disk.rbd0.in‚0 disk.rbd2.in‚0 disk.dm-0.in‚0.078125 disk.dm-0.out‚10465.86328125 disk.sdb.in‚0.03125" proc.memory.rssMB‚ 2893.4296875 disk.sda2.in‚ 0.0390625" disk.nvme1n1.out‚ 4192.98828125 disk.sdb1.in‚0 disk.sda1.in‚0 disk.md0p1.out‚ 117.59375 disk.nvme0n1.in‚ 0.02734375P · “¤‚(:ˆ í‰êÔÓÞ­ÿ gpu.0.gpu‚56.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.195816040039062* gpu.0.memoryAllocatedBytes‚ 14771093504 gpu.0.temp‚39.0 gpu.0.powerWatts‚174.286' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.5715 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0_ͼZ¥‚(:Ï í‰êÔŒæÿ disk.rbd0.out‚ 0.07421875" disk.nvme1n1.out‚ 4193.18359375 disk.nvme0n1.out‚ 6484.015625$ memory_percent‚91.30702018737793( proc.memory.availableMB‚ 1424.2578125 proc.cpu.threads‚17 disk.md0.out‚ 119.5703125$ disk./.usageGB‚2709.590019226074 disk.sda.in‚ 0.0390625 disk.rbd0.in‚0 disk.sdb1.out‚0# proc.memory.rssMB‚ 2893.44921875 disk.rbd2.out‚ 0.16552734375 disk.dm-0.in‚0.078125 disk.md0p1.in‚ 0.0703125* proc.memory.percent‚17.660212516784668 disk.sdb.in‚0.03125 disk.dm-0.out‚10677.19921875 network.sent‚ 1.354986e+06) disk./.usagePercent‚75.76127098908614 disk.md0.in‚ 0.0703125 disk.sdb2.in‚0.03125 disk.sda1.in‚0 disk.md0p1.out‚ 119.5703125 disk.nvme0n1.in‚ 0.02734375 disk.sdb.out‚123.9755859375 disk.sda2.out‚123.9755859375 disk.rbd2.in‚0 disk.sdb2.out‚123.9755859375 disk.sda1.out‚0 cpu‚25.058765167790032 disk.sda.out‚123.9755859375 disk.sda2.in‚ 0.0390625 disk.nvme1n1.in‚ 0.05078125 disk.sdb1.in‚0 network.recv‚426419î|À«’¦‚(:‡ ü‰êÔÔ”»ÿ gpu.0.gpu‚28.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.195816040039062* gpu.0.memoryAllocatedBytes‚ 14771093504 gpu.0.temp‚37.0 gpu.0.powerWatts‚153.176' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚38.294 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ýáì^§‚(:Ó ü‰êÔö¯ disk.dm-0.out‚10726.46484375 disk.nvme0n1.in‚ 0.02734375" disk.nvme0n1.out‚ 6532.91015625 disk.sda.in‚ 0.0390625 disk.sda2.out‚125.5439453125* proc.memory.percent‚17.660212516784668 network.recv‚430822 disk.md0p1.out‚ 121.07421875# memory_percent‚91.3092851638794 proc.cpu.threads‚17 disk.md0.in‚ 0.0703125 disk.sda1.out‚0) disk./.usagePercent‚75.76126245624359 disk.dm-0.in‚0.078125 disk.sdb.in‚0.03125 disk.sda.out‚125.5439453125 disk.rbd0.in‚0 disk.sdb1.out‚0 disk.rbd2.in‚0# proc.memory.rssMB‚ 2893.44921875 cpu‚25.05835709742456$ disk./.usageGB‚2709.589714050293 disk.nvme1n1.in‚ 0.05078125 disk.sdb2.in‚0.03125 disk.sdb.out‚125.5439453125 disk.sda2.in‚ 0.0390625 disk.rbd2.out‚ 0.16552734375 network.sent‚ 1.363077e+06 disk.rbd0.out‚ 0.1533203125 disk.sdb2.out‚125.5439453125 disk.sda1.in‚0) proc.memory.availableMB‚ 1423.88671875 disk.md0.out‚ 121.07421875 disk.md0p1.in‚ 0.0703125! disk.nvme1n1.out‚ 4193.5546875 disk.sdb1.in‚0ÔÕúc2¨- _wandbruntime‚1111  _runtime‚1111m)¯çÅ©¿ & lr-AdamW/pg1‚0.0007087203472332967 & lr-AdamW/pg2‚0.0007087203472332967 trainer/global_step‚999 % _timestamp‚1.7885114856814046e+09 _runtime‚1111.151941637 _step‚76²Ûɪà _step‚76 & lr-AdamW/pg1‚0.0007087203472332967 & lr-AdamW/pg2‚0.0007087203472332967 trainer/global_step‚999 % _timestamp‚1.7885114856814046e+09 _runtime‚1111.151941637L ¢c)2«- _wandbruntime‚1111  _runtime‚1111ðÝY 9¬ÔO IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 916643840 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚12382053 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚18132160 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914443232 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14218690560 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30188 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚80768433193984 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚19908654 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1160848072192 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚629 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚90500392014336 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24592 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚78001654213520 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24592 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚24673075 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24590 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 398533632 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 246679040 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5750105 _runtime‚1111.154064017 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚79655699393024 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 898640896 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1170 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1170 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚79655717395968 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 % _timestamp‚1.7885114856830997e+09 _step‚77 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚19908656 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5749478 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1189 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 225432576 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚78001671252880 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚563 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1176280064 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13954824192 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚44581731 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚24648485 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 249372672 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚80768433193984 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1109472122264 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1108574718392 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚563 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 898640896 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚79655717395968 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚12382055 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14218690560 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚80769349837824 OIDeviceStatsMonü_±þ¡ itor.on_train_batch_start/inactive_split.large_pool.current‚2 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24590 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1112733800960 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚89339520002048 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1174999693 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1180696576 trainer/global_step‚999 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚24673075 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚563 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚19908654 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚80769349837824 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚89339543942144 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1180696576 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30433 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1164 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚90500142641664 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1160622639616 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 916643840 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚19908656 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30188 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚79655699393024 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13954824192 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1113632441856 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1112733800960 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚18131531 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510207117 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚24648485ÙW?Þ'­ØO L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚19908654 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚80769349837824 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚89339543942144 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1180696576 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30433 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1164 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚90500142641664 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1160622639616 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 916643840 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚19908656 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30188 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚79655699393024 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13954824192 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1113632441856 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1112733800960 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚18131531 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510207117 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚24648485 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 916643840 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚12382053 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚18132160 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914443232 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14218690560 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30188 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚80768433193984 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚19908654 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1160848072192 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚629 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚90500392014336 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24592 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚78001654213520 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24592 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚24673075 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24590 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 398533632 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 246679040 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5750105 _runtime‚1111.154064017 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚79655699393024 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 898640896 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1170 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1170 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚79655717395968 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 % _timestamp‚1.7885114856830997e+09 _step‚77 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚19908656 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5749478 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1189 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 225432576 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚78001671252880 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚563 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1176280064 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13954824192 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚44581731 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚24648485 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 249372672 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚80768433193984 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1109472122264 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1108574718392 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚563 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 898640896 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚79655717395968 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚12382055 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14218690560 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚80769349837824 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24590 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1112733800960 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚89339520002048 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1174999693 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1180696576 trainer/global_step‚999 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚24673075 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚563ML'hÀ2®- _wandbruntime‚1112  _runtime‚1112`å"ä&¯ÞM S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚80850092950016 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1109683379710 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14218690560 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1176280064 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚12394431 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚12394429 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1164 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚24697692 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚18150268 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚18149641 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5755837 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚563 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚80850092950016 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 898654208 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚79735347691520 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1161796173312 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1170 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24616 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1113846604288 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚90590880561152 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 245165056 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1170 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 916657152 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚625 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚90590635396096 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1189 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚89428839222784 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1180696576 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚24673076 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 246679040 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚79190232130798 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510207117 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14218690560 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚89428863162880 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30433 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚44601628 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚79189317687358 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1180696576 _step‚78 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚19928552 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30188 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚19928552 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1114745258496 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚627 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1170 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚24673076 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚44626246 % _timestamp‚1.7885114869580529e+09 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24618 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 221224960 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1110580783790 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1113846604288 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13954824192 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1176280064 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30433 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 trainer/global_step‚999 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚79735329688576 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1114745258496 ODeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13954824192 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1176280064 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30433 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 trainer/global_step‚999 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚79735329688576 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1114745258496 ODeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚18149641 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5755837 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚563 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚80850092950016 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 898654208 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚79735347691520 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1161796173312 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1170 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24616 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1113846604288 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚90590880561152 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 245165056 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1170 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 916657152 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚625 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚90590635396096 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1189 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚89428839222784 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1180696576 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚24673076 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 246679040 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚79190232130798 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510207117 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14218690560 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚89428863162880 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30433 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚44601628 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚79189317687358 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1180696576 _step‚78 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚19928552 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30188 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚19928552 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1114745258496 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚627 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1170 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚24673076 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚44626246 % _timestamp‚1.7885114869580529e+09 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24618 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 221224960 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 11a)m 10580783790 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1113846604288 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984NÔWZã2±- _wandbruntime‚1112  _runtime‚1112­$N¼S²Í _runtime‚1112.432674038 % _timestamp‚1.7885114869622042e+09 0loss/FBt4/assignment_loss‚0.7060290746953797 /loss/FRt2/assignment_loss‚1.950337986762707 0loss/FRt3/assignment_loss‚3.9048182887059673 2loss/SRqqt2/assignment_loss‚1.1320802759911333 2loss/SRqqt4/assignment_loss‚2.6864515352249145 0loss/SRqqt3/detection_loss‚1.781415009405464 0loss/SRqqt4/detection_loss‚2.699979816675186 1loss/FBt2/assignment_loss‚0.20932232724551558 /loss/FRt1/detection_loss‚2.2567173619019356 1loss/SRqqt2/detection_loss‚1.0538062865712814 trainer/global_step‚999 epoch‚0 0loss/FBt3/assignment_loss‚0.2979416251182556 0loss/SRqqt1/detection_loss‚2.595087049468871 %loss/total_loss‚53.01558996646273 /loss/FBt2/detection_loss‚0.6587534151705248 .loss/FBt4/detection_loss‚2.901286161862887 /loss/FRt4/assignment_loss‚4.903875224292278 0loss/FBt1/assignment_loss‚0.7184610497206445 _step‚79 /loss/FRt1/assignment_loss‚4.387603339396025 .loss/FRt2/detection_loss‚1.794186097383499 /loss/FBt1/detection_loss‚2.6384879536926746 1loss/SRqqt1/assignment_loss‚2.177620517630731 2loss/SRqqt3/assignment_loss‚2.1657727188430727 .loss/FRt3/detection_loss‚2.323918553414168 /loss/FRt4/detection_loss‚2.7829420678317547 .loss/FBt3/detection_loss‚4.288696229457855©b‡°W³Ñ epoch‚0 0loss/FBt3/assignment_loss‚0.2979416251182556 0loss/SRqqt1/detection_loss‚2.595087049468871 %loss/total_loss‚53.01558996646273 /loss/FBt2/detection_loss‚0.6587534151705248 .loss/FBt4/detection_loss‚2.901286161862887 /loss/FRt4/assignment_loss‚4.903875224292278 0loss/FBt1/assignment_loss‚0.7184610497206445 _step‚79 /loss/FRt1/assignment_loss‚4.387603339396025 .loss/FRt2/detection_loss‚1.794186097383499 /loss/FBt1/detection_loss‚2.6384879536926746 1loss/SRqqt1/assignment_loss‚2.177620517630731 2loss/SRqqt3/assignment_loss‚2.1657727188430727 .loss/FRt3/detection_loss‚2.323918553414168 /loss/FRt4/detection_loss‚2.7829420678317547 .loss/FBt3/detection_loss‚4.288696229457855 _runtime‚1112.432674038 % _timestamp‚1.7885114869622042e+09 0loss/FBt4/assignment_loss‚0.7060290746953797 /loss/FRt2/assignment_loss‚1.950337986762707 0loss/FRt3/assignment_loss‚3.9048182887059673 2loss/SRqqt2/assignment_loss‚1.1320802759911333 2loss/SRqqt4/assignment_loss‚2.6864515352249145 0loss/SRqqt3/detection_loss‚1.781415009405464 0loss/SRqqt4/detection_loss‚2.699979816675186 1loss/FBt2/assignment_loss‚0.20932232724551558 /loss/FRt1/detection_loss‚2.2567173619019356 1loss/SRqqt2/detection_loss‚1.0538062865712814 trainer/global_step‚999OÕØ;½`´‚(:Õ ‹ŠêÔ¼„÷þ disk.rbd0.out‚ 0.1533203125 disk.sdb2.in‚0.03125 disk.dm-0.out‚ 10728.265625 disk.sda1.in‚0 disk.nvme0n1.in‚ 0.02734375 disk.sdb.in‚0.03125 disk.sda.in‚ 0.0390625# proc.memory.rssMB‚ 2893.44921875 disk.sdb1.out‚0 disk.md0.in‚ 0.0703125 disk.sdb.out‚126.95947265625 disk.sda2.in‚ 0.0390625! disk.sda2.out‚126.95947265625% disk./.usageGB‚2709.5290451049805! disk.nvme1n1.out‚ 4193.7890625 cpu‚25.057489499253887 disk.rbd0.in‚0 disk.md0.out‚ 122.44140625 disk.md0p1.out‚ 122.44140625) proc.memory.availableMB‚ 1424.52734375 proc.cpu.threads‚17 network.recv‚434011 disk.rbd2.in‚0! disk.sdb2.out‚126.95947265625 disk.sda.out‚126.95947265625 disk.sdb1.in‚0 disk.sda1.out‚0 disk.md0p1.in‚ 0.0703125! disk.nvme0n1.out‚ 6534.4765625$ memory_percent‚91.30537509918213 disk.nvme1n1.in‚ 0.05078125 disk.rbd2.out‚ 0.228515625 disk.dm-0.in‚0.078125* proc.memory.percent‚17.660212516784668 network.sent‚ 1.365827e+06) disk./.usagePercent‚75.75956612714072¬ø3K”µ‚(:‰ ‹ŠêÔÿ«Ê€ gpu.0.gpu‚56.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.195816040039062* gpu.0.memoryAllocatedBytes‚ 14771093504 gpu.0.temp‚39.0 gpu.0.powerWatts‚170.559' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.63975 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0"Δ¶‚(:‰ šŠêÔµçñÿ gpu.0.gpu‚40.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.195816040039062* gpu.0.memoryAllocatedBytes‚ 14771093504 gpu.0.temp‚38.0 gpu.0.powerWatts‚154.735' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚38.68375 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0»%•Ç[·‚(:Ð šŠêÔ¶¯ÿ€ disk.rbd2.out‚ 0.228515625 disk.dm-0.out‚ 10878.9375 disk.md0p1.out‚ 124.421875" disk.nvme0n1.out‚ 6614.22265625 cpu‚25.057085838912283 network.sent‚ 1.415221e+06 disk.sda2.in‚ 0.0390625 disk.sdb.in‚0.03125 disk.dm-0.in‚0.078125 network.recv‚439111 disk.rbd0.out‚ 0.21630859375 disk.md0.in‚ 0.0703125 disk.rbd2.in‚0 disk.sda1.out‚0' proc.memory.availableMB‚ 1424.453125 disk.sda.in‚ 0.0390625! disk.sda2.out‚129.00439453125 disk.nvme1n1.in‚ 0.05078125 disk.sdb2.in‚0.03125 disk.sda1.in‚0 disk.md0p1.in‚ 0.0703125 disk.rbd0.in‚0" disk.nvme1n1.out‚ 4264.71484375 disk.nvme0n1.in‚ 0.02734375 disk.sdb.out‚129.00439453125$ memory_percent‚91.30582809448242) disk./.usagePercent‚75.76349315461238% disk./.usageGB‚2709.6694946289062 disk.sdb1.out‚0 disk.md0.out‚ 124.421875 proc.cpu.threads‚17 disk.sda.out‚129.00439453125 disk.sdb1.in‚0! disk.sdb2.out‚129.00439453125# proc.memory.rssMB‚ 2893.44921875* proc.memory.percent‚17.6602125167846687*“˜d¸‚(:Ù ©ŠêÔ³ùÁ€ disk.rbd0.out‚ 0.21630859375" disk.nvme1n1.out‚ 4335.62109375 disk.sda.out‚130.62255859375 disk.rbd0.in‚0) proc.memory.availableMB‚ 1424.33203125 cpu‚25.056593260097003% disk./.usageGB‚2709.7982749938965 disk.sdb1.out‚0 disk.rbd2.in‚0 disk.rbd2.out‚ 0.29150390625 network.sent‚ 1.422656e+06 disk.sda1.in‚0 disk.sda1.out‚0 disk.md0p1.in‚ 0.0703125 disk.nvme0n1.in‚ 0.02734375$ memory_percent‚91.30656719207764! disk.sda2.out‚130.62255859375 disk.nvme1n1.in‚ 0.05078125 disk.md0.out‚ 125.9765625 disk.sdb.in‚0.03125 disk.sda2.in‚ 0.0390625# proc.memory.rssMB‚ 2893.44921875 disk.sdb2.in‚0.03125 disk.dm-0.in‚0.078125 disk.sda.in‚ 0.0390625* proc.memory.percent‚17.660212516784668 proc.cpu.threads‚17" disk.nvme0n1.out‚ 6700.82421875 disk.sdb.out‚130.62255859375 network.recv‚442362) disk./.usagePercent‚75.76709390751627 disk.sdb1.in‚0 disk.md0.in‚ 0.0703125! disk.sdb2.out‚130.62255859375 disk.md0p1.out‚ 125.9765625 disk.dm-0.out‚ 11036.4453125ôÂù#”¹‚(:‰ ©ŠêÔÑïÓ€ gpu.0.gpu‚52.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.195816040039062* gpu.0.memoryAllocatedBytes‚ 14771093504 gpu.0.temp‚38.0 gpu.0.powerWatts‚168.383' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.09575 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0±ybu2º- _wandbruntime‚1167  _runtime‚1167Å4NöÅ»¿ & lr-AdamW/pg1‚0.0007441918360838119 & lr-AdamW/pg2‚0.0007441918360838119 trainer/global_step‚1049 % _timestamp‚1.7885115416965754e+09 _runtime‚ 1167.16696645 _step‚80­#ɼà _runtime‚ 1167.16696645 _step‚80 & lr-AdamW/pg1‚0.0007441918360838119 & lr-AdamW/pg2‚0.0007441918360838119 trainer/global_step‚1049 % _timestamp‚1.7885115416965754e+09PJ¸o2½- _wandbruntime‚1167  _runtime‚1167“ƒ¯Û'¾ÕO LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚563 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30188 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚81900658919920 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1170 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚84805602551808 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6038469 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚93805504978944 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1169287022080 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚626 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510207117 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7885115416973317e+09 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24592 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚46784941 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚25905977 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 246679040 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14218690560 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1180696576 _runtime‚1167.167869994 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚83637214170624 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1176280064 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 898640896 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30433 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1180696576 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚83637214170624 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚93805481038848 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1169287022080 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚83064680295420 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1164021375500 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1176280064 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚624 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24590 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚83637232173568 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚25881387 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚95024211600384 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 916643840 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚84805602551808 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚84806519195648 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 916643840 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6037845 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30433 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1164918779372 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚20903556 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚13000855 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897403872 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚13000853 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚563 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1168388381184 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1218730561536 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1170 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1170 trainer/global_step‚1049 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30188 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1191 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚20903554 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚19038698 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚95024460973056 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1218955994112 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚81900675959280 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1180696576 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚563 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚83637232173568 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚20903554 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1166 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚83065594738652 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚20903556 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13954824192 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1170 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24590 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 225432576 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14218690560 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚84806519195648 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚25881387 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1168388381184 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚46809533 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24592 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 249372672 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 _step‚81 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13954824192 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚19039324 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914443232 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14218690560@Z…öß'¿ÙO H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30188 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1191 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚20903554 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚19038698 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚95024460973056 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1218955994112 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚81900675959280 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1180696576 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚563 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚83637232173568 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚20903554 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1166 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚83065594738652 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚20903556 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13954824192 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1170 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24590 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 225432576 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14218690560 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚84806519195648 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚25881387 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1168388381184 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚46809533 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24592 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 249372672 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 _step‚81 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13954824192 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚19039324 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914443232 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14218690560 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚563 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30188 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚81900658919920 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1170 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚84805602551808 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6038469 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚93805504978944 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1169287022080 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚626 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510207117 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7885115416973317e+09 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24592 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚46784941 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚25905977 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 246679040 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14218690560 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1180696576 _runtime‚1167.167869994 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚83637214170624 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1176280064 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 898640896 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30433 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1180696576 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚83637214170624 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚93805481038848 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1169287022080 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚83064680295420 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1164021375500 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1176280064 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚624 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24590 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚83637232173568 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚25881387 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚95024211600384 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 916643840 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚84805602551808 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚84806519195648 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 916643840 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6037845 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30433 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1164918779372 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚20903556 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚13000855 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897403872 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚13000853 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚563 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1168388381184 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1218730561536 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1170 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1170 trainer/global_step‚1049Q‰8¹€2À- _wandbruntime‚1168  _runtime‚11680üÞsMÁÞM E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30433 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 916657152 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1219914962944 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 246679040 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚46829430 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚25905978 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚46854048 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚20923454 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30188 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚83143769050866 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚563 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚20923452 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚83144683494306 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1180696576 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚25930594 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚83716844466176 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6043534 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 398533632 % _timestamp‚1.7885115427125344e+09 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚81978656053408 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1174999693 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 898654208 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚46854048 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚46829430 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1169501184512 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚25930594 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚25905978 trainer/global_step‚1049 _step‚82 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1169501184512 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚81978639014048 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 898654208 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1176280064 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚563 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6044133 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚84887262307840 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.la:Œ^ˆ—rge_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚601 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14218690560 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 210739200 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚13013229 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚83716862469120 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30433 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914443440 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1180696576 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24616 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1191 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 234679296 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1170 _runtime‚ 1168.18367869 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚19056763 ODeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13954824192 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510207117 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1166027440898 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1170399838720 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚95114949901824 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚563 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1170 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13954824192 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510207117 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1166027440898 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1170399838720 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚95114949901824 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚563 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1170 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 916657152 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1219914962944 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 246679040 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚46829430 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚25905978 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚46854048 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚20923454 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30188 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚83143769050866 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚563 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚20923452 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚83144683494306 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1180696576 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚25930594 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚83716844466176 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6043534 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 398533632 % _timestamp‚1.7885115427125344e+09 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚81978656053408 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1174999693 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 898654208 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚46854048 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚46829430 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1169501184512 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚25930594 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚25905978 trainer/global_step‚1049 _step‚82 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1169501184512 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚81978639014048 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 898654208 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1176280064 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚563 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6044133 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚84887262307840 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚601 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14218690560 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 210739200 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚13013229 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚83716862469120 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30433 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914443440 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1180696576 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24616 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1191 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 234679296 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1170 _runtime‚ 1168.18367869 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚19056763 ODeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚88842771909632 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1276836417024 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚87618728948224 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 246679040 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 916643840 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚19947946 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 898640896 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1176280064 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6328910 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30188 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24592 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚98271466015744 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚87020046102160 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914443232 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24590 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 H?DeviceStatsMonitor.on_train_batch_start/inactive_spli1(„­[ t.all.peak‚1191 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1166 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚85799680665680 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚27138879 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30188 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30433 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚87618728948224 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚87618746951168 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1220365436480 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚21898456 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚619 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1224042961408 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 398533632 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1174999693 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1224042961408 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚88843688553472 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13954824192 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚27114289 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1224941602304 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 898640896 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚27138879 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171cQŸß'ÓÙO PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚27138879 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30188 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30433 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚87618728948224 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚87618746951168 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1220365436480 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚21898456 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚619 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1224042961408 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 398533632 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1174999693 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1224042961408 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚88843688553472 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13954824192 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚27114289 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1224941602304 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 898640896 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚27138879 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚21898456 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1219468032608 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13954824192 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24590 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 trainer/global_step‚1099 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚88842771909632 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1276836417024 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚87618728948224 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 246679040 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 916643840 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚19947946 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 898640896 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1176280064 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6328910 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30188 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24592 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚98271466015744 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚87020046102160 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914443232 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚24590 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1191 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1166 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚85799680665680 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0Umí~é2Ô- _wandbruntime‚1227  _runtime‚12276æêRå&ÕßM N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13954824192 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚21918354 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24616 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚27163496 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚98360785236480 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510207117 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚49057232 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚636 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚98360761296384 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6333960 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚49081850 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚49057232 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 _step‚86 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚21918354 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30433 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚21918352 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1191 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚99638744016896 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 231710720 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914443440 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1221474098006 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1277982720512 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚87098220414374 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1225155764736 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚87099134857814 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 916657152 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚13632029 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1226054418944 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 % _timestamp‚1.7885116020115979e+09 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13954824192 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 916657152 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1166 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30433 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚19965989 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1226054418944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1176280064 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 118279372823¤Œ%ÖãM @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6333960 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚49081850 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚49057232 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 _step‚86 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚21918354 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30433 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚21918352 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1191 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚99638744016896 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 231710720 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914443440 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1221474098006 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1277982720512 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚87098220414374 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1225155764736 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚87099134857814 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 916657152 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚13632029 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1226054418944 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 % _timestamp‚1.7885116020115979e+09 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13954824192 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 916657152 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1166 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30433 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚19965989 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1226054418944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1176280064 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13954824192 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚21918354 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24616 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚27163496 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚98360785236480 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510207117 C6DeviceStatsMonitor.on_]^ë]train_batch_end/active.all.freed‚49057232 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚636 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚98360761296384Všç£2×- _wandbruntime‚1227  _runtime‚1227Þ%{AVØÐ /loss/FBt4/detection_loss‚2.9729646096626916 0loss/FBt1/assignment_loss‚0.6308540219538971 2loss/FBt3/assignment_loss‚0.050430115312337875 1loss/SRqqt3/detection_loss‚1.7686507914398537 %loss/total_loss‚52.72260746172137 % _timestamp‚1.7885116020140705e+09 _step‚87 0loss/FRt2/assignment_loss‚1.9422087356692455 .loss/FRt4/assignment_loss‚4.96444022270941 1loss/SRqqt1/assignment_loss‚2.116976635578351 0loss/FBt2/assignment_loss‚0.2380959167299398 /loss/FRt2/detection_loss‚1.6627811804169514 1loss/SRqqt1/detection_loss‚2.4983860506461215 /loss/FBt1/detection_loss‚2.3293882702526294 /loss/FBt3/detection_loss‚4.5645856857299805 trainer/global_step‚1099 _runtime‚ 1227.48418762 .loss/FRt1/assignment_loss‚4.23407378130489 .loss/FRt3/detection_loss‚2.634713880040429 0loss/SRqqt2/detection_loss‚1.213215293924487 2loss/SRqqt3/assignment_loss‚2.2843720385285673 /loss/FRt1/detection_loss‚2.7312799195448556 epoch‚0 2loss/SRqqt2/assignment_loss‚1.1172743148155628 .loss/FRt4/detection_loss‚3.007102281816544 0loss/SRqqt4/detection_loss‚2.230921999404305 /loss/FBt2/detection_loss‚0.6861677015133735 0loss/FRt3/assignment_loss‚3.7768327824094077 2loss/SRqqt4/assignment_loss‚2.3341991348486197 0loss/FBt4/assignment_loss‚0.7326920974689225±[kEZÙÔ 2loss/SRqqt2/assignment_loss‚1.1172743148155628 .loss/FRt4/detection_loss‚3.007102281816544 0loss/SRqqt4/detection_loss‚2.230921999404305 /loss/FBt2/detection_loss‚0.6861677015133735 0loss/FRt3/assignment_loss‚3.7768327824094077 2loss/SRqqt4/assignment_loss‚2.3341991348486197 0loss/FBt4/assignment_loss‚0.7326920974689225 /loss/FBt4/detection_loss‚2.9729646096626916 0loss/FBt1/assignment_loss‚0.6308540219538971 2loss/FBt3/assignment_loss‚0.050430115312337875 1loss/SRqqt3/detection_loss‚1.7686507914398537 %loss/total_loss‚52.72260746172137 % _timestamp‚1.7885116020140705e+09 _step‚87 0loss/FRt2/assignment_loss‚1.9422087356692455 .loss/FRt4/assignment_loss‚4.96444022270941 1loss/SRqqt1/assignment_loss‚2.116976635578351 0loss/FBt2/assignment_loss‚0.2380959167299398 /loss/FRt2/detection_loss‚1.6627811804169514 1loss/SRqqt1/detection_loss‚2.4983860506461215 /loss/FBt1/detection_loss‚2.3293882702526294 /loss/FBt3/detection_loss‚4.5645856857299805 trainer/global_step‚1099 _runtime‚ 1227.48418762 .loss/FRt1/assignment_loss‚4.23407378130489 .loss/FRt3/detection_loss‚2.634713880040429 0loss/SRqqt2/detection_loss‚1.213215293924487 2loss/SRqqt3/assignment_loss‚2.2843720385285673 /loss/FRt1/detection_loss‚2.7312799195448556 epoch‚0Wÿ†ê”Ú‚(:‰ ôŠêÔÛ„æÿ gpu.0.gpu‚40.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚37.0 gpu.0.powerWatts‚126.829' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚31.70725 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0-`ÙVÛ‚(:Ë ôŠêÔµú‡‚ disk.sdb2.in‚0.03125 disk.sdb.in‚0.03125) disk./.usagePercent‚75.79156343349298 disk.sda1.out‚0* proc.memory.percent‚17.660212516784668 disk.sdb1.out‚0 disk.rbd2.out‚ 0.29150390625 disk.sda1.in‚0 disk.dm-0.out‚ 11946.421875" disk.nvme0n1.out‚ 7105.41015625 cpu‚25.053347286297914 proc.cpu.threads‚17 disk.nvme1n1.in‚ 0.05078125 disk.md0.out‚ 134.1640625 disk.dm-0.in‚0.078125 disk.md0p1.out‚ 134.1640625 network.sent‚ 1.50624e+06# disk./.usageGB‚2710.67342376709 disk.sda2.out‚ 139.115234375 disk.sdb1.in‚0 disk.rbd2.in‚0 disk.sdb.out‚ 139.115234375 network.recv‚463571 disk.sda.in‚ 0.0390625$ memory_percent‚91.30980968475342# proc.memory.rssMB‚ 2893.44921875 disk.sda.out‚ 139.115234375" disk.nvme1n1.out‚ 4841.01171875 disk.md0.in‚ 0.0703125 disk.sdb2.out‚ 139.115234375) proc.memory.availableMB‚ 1423.80078125 disk.sda2.in‚ 0.0390625 disk.md0p1.in‚ 0.0703125 disk.nvme0n1.in‚ 0.02734375 disk.rbd0.in‚0 disk.rbd0.out‚ 0.279296875¾ëˆ”Ü‚(:‰ ƒ‹êÔ§Ö»ÿ gpu.0.gpu‚57.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚162.481' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚40.62025 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0[Mõ-`Ý‚(:Õ ƒ‹êÔ»Ìóÿ) disk./.usagePercent‚75.79150583680563! disk.nvme1n1.out‚ 4841.1328125 disk.sdb2.in‚0.03125 disk.dm-0.in‚0.078125) proc.memory.availableMB‚ 1423.69140625! disk.sdb2.out‚141.15478515625 disk.rbd0.out‚ 0.279296875 disk.nvme1n1.in‚ 0.05078125 disk.sda.in‚ 0.0390625 disk.sda.out‚141.15478515625 disk.rbd0.in‚0 disk.rbd2.out‚ 0.29150390625* proc.memory.percent‚17.660212516784668 disk.dm-0.out‚ 11946.609375 disk.md0p1.out‚ 136.14453125 cpu‚25.052806277895996 disk.sda2.in‚ 0.0390625 disk.md0.in‚ 0.0703125 disk.md0.out‚ 136.14453125 proc.cpu.threads‚17! disk.sda2.out‚141.15478515625 disk.sdb1.in‚0 disk.sda1.out‚0 disk.nvme0n1.in‚ 0.02734375 disk.sda1.in‚0 disk.md0p1.in‚ 0.0703125 network.sent‚ 1.542384e+06% disk./.usageGB‚2710.6713638305664# memory_percent‚91.3104772567749 network.recv‚468073 disk.sdb1.out‚0! disk.nvme0n1.out‚ 7105.4765625# proc.memory.rssMB‚ 2893.44921875 disk.sdb.in‚0.03125 disk.sdb.out‚141.15478515625 disk.rbd2.in‚0C]*”Þ‚(:‰ ’‹êÔµ¾³ÿ gpu.0.gpu‚31.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚164.989' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚41.24725 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0vî…SDß‚(:¹ ’‹êÔŽŽæ€ disk.nvme0n1.in‚ 0.02734375 disk.sda2.out‚ 142.67578125* proc.memory.percent‚17.660212516784668 disk.sdb.out‚ 142.67578125 disk.sda.out‚ 142.67578125 disk.nvme1n1.in‚ 0.05078125 disk.md0.out‚137.625 disk.sda1.in‚0% disk./.usageGB‚2710.1438522338867 disk.nvme0n1.out‚ 7307.5625 disk.rbd0.out‚ 0.279296875 disk.sdb1.in‚0 disk.dm-0.in‚0.078125 disk.md0p1.in‚ 0.0703125 network.sent‚ 1.550844e+06 network.recv‚472691 disk.sda2.in‚ 0.0390625 disk.rbd2.out‚ 0.29150390625 disk.sdb2.out‚ 142.67578125# proc.memory.rssMB‚ 2893.44921875 disk.rbd0.in‚0 disk.sda1.out‚0 disk.dm-0.out‚ 12448.921875 disk.sdb.in‚0.03125 disk.nvme1n1.out‚ 5141.359375( proc.memory.availableMB‚ 1423.4453125 cpu‚25.05215381166912 proc.cpu.threads‚17( disk./.usagePercent‚75.7767563917817 disk.sda.in‚ 0.0390625 disk.rbd2.in‚0$ memory_percent‚91.31197929382324 disk.sdb1.out‚0 disk.md0.in‚ 0.0703125 disk.sdb2.in‚0.03125 disk.md0p1.out‚137.625\–œ[’à‚(:‡ ¡‹êÔˆÃÿ gpu.0.gpu‚39.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚36.0 gpu.0.powerWatts‚121.536' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚30.384 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0o¶LåTá‚(:É ¡‹êÔ¬— disk.dm-0.out‚12597.69140625 disk.md0p1.out‚139.25! disk.nvme0n1.out‚ 7356.1953125 disk.sdb.in‚0.03125 disk.nvme1n1.in‚ 0.05078125! disk.sdb2.out‚144.35595703125 disk.sda.in‚ 0.0390625 cpu‚25.051728546839% disk./.usageGB‚2710.1435585021973 disk.sda1.in‚0! disk.sda2.out‚144.35595703125 network.recv‚476298 disk.md0.out‚139.25 disk.rbd2.in‚0 disk.rbd2.out‚ 0.29150390625) proc.memory.availableMB‚ 1423.80859375 proc.cpu.threads‚17) disk./.usagePercent‚75.77674817892074 disk.md0p1.in‚ 0.0703125 disk.nvme0n1.in‚ 0.02734375 disk.sda.out‚144.35595703125 disk.rbd0.in‚0 disk.rbd0.out‚ 0.279296875" disk.nvme1n1.out‚ 5241.49609375 disk.md0.in‚ 0.0703125 network.sent‚ 1.556107e+06 disk.dm-0.in‚0.078125 disk.sdb2.in‚0.03125# memory_percent‚91.3097620010376# proc.memory.rssMB‚ 2893.44921875* proc.memory.percent‚17.660212516784668 disk.sda1.out‚0 disk.sda2.in‚ 0.0390625 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.sdb.out‚144.35595703125b–ä2â- _wandbruntime‚1285  _runtime‚1285_ÈG~ÆãÀ _runtime‚1285.650985348 _step‚88 & lr-AdamW/pg1‚0.0008151348137848427 & lr-AdamW/pg2‚0.0008151348137848427 trainer/global_step‚1149 % _timestamp‚1.7885116601806757e+09xQ•5ÊäÄ trainer/global_step‚1149 % _timestamp‚1.7885116601806757e+09 _runtime‚1285.650985348 _step‚88 & lr-AdamW/pg1‚0.0008151348137848427 & lr-AdamW/pg2‚0.0008151348137848427X™WŽ 2å- _wandbruntime‚1285  _runtime‚1285ú„ÙÃß'æÙO JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚639 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚637 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30433 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚51265137 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚22893356 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24590 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚90974497465668 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1191 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚104072465578496 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚28347191 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚102737403112448 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚92879941267456 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24592 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 246679040 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1274914689716 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24592 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚91600261728768 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1280596182528 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 _runtime‚1285.652108153 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30188 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1280596182528 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚102737427052544 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 916643840 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897403872 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1279697541632 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13954824192 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 898640896 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚89698685372080 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1174999693 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚22893356 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6618762 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 398533632 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚92879941267456 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚92880857911296 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30433 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1166 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1275812093588 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚91600243725824 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚104072710756864 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1335062466048 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚14238453 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914443232 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚28371781 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚91600261728768 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚91600243725824 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510207117 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚22893354 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚28347191 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚28371781 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 % _timestamp‚1.7885116601816573e+09 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 916643840 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚22893354 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30188 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728Óã'çÝO L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚22893354 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚28347191 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚28371781 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 % _timestamp‚1.7885116601816573e+09 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 916643840 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚22893354 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30188 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚639 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚637 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30433 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚51265137 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚22893356 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24590 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚90974497465668 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1191 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚104072465578496 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚28347191 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚102737403112448 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚92879941267456 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24592 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 246679040 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1274914689716 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚24592 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚91600261728768 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1280596182528 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 _runtime‚1285.652108153 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30188 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1280596182528 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚102737427052544 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 916643840 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897403872 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1279697541632 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13954824192 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 898640896 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚89698685372080 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1174999693 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚22893356 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6618762 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 398533632 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚92879941267456 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚92880857911296 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30433 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1166 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1275812093588 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚91600243725824 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚104072710756864 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1335062466048 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚14238453 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914443232 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚28371781 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚91600261728768 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚91600243725824 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510207117 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0Yk¿bC2è- _wandbruntime‚1286  _runtime‚1286Ï2Y)± éãM C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30433 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚22913252 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚14250829 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚91052671777882 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚22913252 ODeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚51309652 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚14250831 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1336233991680 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 _runtime‚1286.778241415 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚91679892024320 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚91053586221322 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510207117 _step‚90 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1191 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1176280064 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13954824192 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897404080 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚20874773 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚89776665466208 % _timestamp‚1.7885116613072197e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30433 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13954824192 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚646 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1336455216640 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚20875419 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1276920755114 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 221224960 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1280810344960 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚91679874021376 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 898654208 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1280810344960 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6624588 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24618 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 916657152 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚644 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚28371782 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚28371782 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚91679892024320 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚104163201489920 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚92961601023488 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1166 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚104162956324864 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚28396398 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1281708999168 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚51285034 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24616 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 398533632 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564rŸü!í&êçM VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24616 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 398533632 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30433 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚22913252 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚14250829 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚91052671777882 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚22913252 ODeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚51309652 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚14250831 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1336233991680 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 _runtime‚1286.778241415 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚91679892024320 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚91053586221322 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510207117 _step‚90 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1191 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1176280064 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13954824192 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897404080 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚20874773 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚89776665466208 % _timestamp‚1.7885116613072197e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚30433 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13954824192 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚646 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1336455216640 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚20875419 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1276920755114 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 221224960 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1280810344960 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚91679874021376 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 898654208 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1280810344960 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6624588 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24618 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 916657152 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚644 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚28371782 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚28371782 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚91679892024320 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚104163201489920 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚92961601023488 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1166 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚104162956324864 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚28396398 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1281708999168 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚51285034ZÈû 2ë- _wandbruntime‚1286  _runtime‚1286NkX›TìÎ /loss/FRt3/assignment_loss‚3.826974584410588 .loss/FRt4/assignment_loss‚5.31685895304526 2loss/SRqqt1/assignment_loss‚1.7628192267305143 _runtime‚1286.780432236 _step‚91 /loss/FRt1/assignment_loss‚4.969769057773409 0loss/FBt1/assignment_loss‚0.6032532863281782 .loss/FBt2/detection_loss‚0.697276869009054 .loss/FRt4/detection_loss‚2.968614685919977 epoch‚0 trainer/global_step‚1149 1loss/FBt2/assignment_loss‚0.24276817971105336 1loss/FBt3/assignment_loss‚0.12777294218540192 /loss/FRt1/detection_loss‚2.3133472062292553 1loss/SRqqt2/detection_loss‚0.9581685477673118 /loss/FBt1/detection_loss‚2.7442863633235297 % _timestamp‚1.7885116613103075e+09 2loss/SRqqt2/assignment_loss‚1.1395782739924851 /loss/FRt3/detection_loss‚2.1752603692002594 .loss/FBt3/detection_loss‚6.358040928840637 %loss/total_loss‚54.10106899001295 0loss/FBt4/assignment_loss‚0.1464627981185913 0loss/SRqqt1/detection_loss‚2.313858794199454 0loss/FRt2/assignment_loss‚1.9479951055682436 1loss/SRqqt4/assignment_loss‚2.565629170129174 /loss/FRt2/detection_loss‚1.9296541773543066 1loss/SRqqt3/detection_loss‚1.7754884258740478 /loss/SRqqt4/detection_loss‚2.38026025891304 .loss/FBt4/detection_loss‚2.897341563151433 2loss/SRqqt3/assignment_loss‚1.939589222237743840x¹XíÒ /loss/FRt1/assignment_loss‚4.969769057773409 0loss/FBt1/assignment_loss‚0.6032532863281782 .loss/FBt2/detection_loss‚0.697276869009054 .loss/FRt4/detection_loss‚2.968614685919977 epoch‚0 trainer/global_step‚1149 1loss/FBt2/assignment_loss‚0.24276817971105336 1loss/FBt3/assignment_loss‚0.12777294218540192 /loss/FRt1/detection_loss‚2.3133472062292553 1loss/SRqqt2/detection_loss‚0.9581685477673118 /loss/FBt1/detection_loss‚2.7442863633235297 % _timestamp‚1.7885116613103075e+09 2loss/SRqqt2/assignment_loss‚1.1395782739924851 /loss/FRt3/detection_loss‚2.1752603692002594 .loss/FBt3/detection_loss‚6.358040928840637 %loss/total_loss‚54.10106899001295 0loss/FBt4/assignment_loss‚0.1464627981185913 0loss/SRqqt1/detection_loss‚2.313858794199454 0loss/FRt2/assignment_loss‚1.9479951055682436 1loss/SRqqt4/assignment_loss‚2.565629170129174 /loss/FRt2/detection_loss‚1.9296541773543066 1loss/SRqqt3/detection_loss‚1.7754884258740478 /loss/SRqqt4/detection_loss‚2.38026025891304 .loss/FBt4/detection_loss‚2.897341563151433 2loss/SRqqt3/assignment_loss‚1.9395892222377438 /loss/FRt3/assignment_loss‚3.826974584410588 .loss/FRt4/assignment_loss‚5.31685895304526 2loss/SRqqt1/assignment_loss‚1.7628192267305143 _runtime‚1286.780432236 _step‚91[…–àK’î‚(:‡ °‹êÔ—ôÿ gpu.0.gpu‚47.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚37.0 gpu.0.powerWatts‚114.408' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.602 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Œiï_ï‚(:Ô °‹êÔ‚‡¦ network.recv‚480752 disk.sdb.in‚0.03125 disk.rbd2.out‚ 0.29150390625 disk.dm-0.in‚0.078125 disk.sdb1.in‚0 disk.sdb2.out‚146.3525390625 disk.sda1.out‚0 proc.cpu.threads‚17 disk.sdb1.out‚0 disk.md0.in‚ 0.0703125 disk.md0.out‚ 141.18359375 disk.md0p1.in‚ 0.0703125( proc.memory.availableMB‚ 1423.4140625) disk./.usagePercent‚75.77748349662919" disk.nvme0n1.out‚ 7356.31640625 disk.sdb.out‚146.3525390625" disk.nvme1n1.out‚ 5242.31640625 network.sent‚ 1.560007e+06 disk.nvme0n1.in‚ 0.02734375 disk.sda.in‚ 0.0390625 disk.sda.out‚146.3525390625 disk.sda2.in‚ 0.0390625 disk.rbd2.in‚0# proc.memory.rssMB‚ 2893.44921875 disk.sdb2.in‚0.03125 disk.sda1.in‚0 disk.dm-0.out‚ 12598.6328125$ memory_percent‚91.31217002868652 cpu‚25.051134655784292 disk.md0p1.out‚ 141.18359375* proc.memory.percent‚17.660212516784668% disk./.usageGB‚2710.1698570251465 disk.sda2.out‚146.3525390625 disk.rbd0.in‚0 disk.rbd0.out‚ 0.279296875 disk.nvme1n1.in‚ 0.05078125<’tá“ð‚(:ˆ ¿‹êÔÖÄÌÿ gpu.0.gpu‚24.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚174.562' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.6405 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0RfõSXñ‚(:Í ¿‹êÔøƒçÿ network.recv‚485626 disk.md0.out‚ 142.7421875! disk.sdb2.out‚147.98095703125 disk.sda1.in‚0 disk.sda.out‚147.98095703125 disk.sdb1.in‚0 disk.sdb2.in‚0.03125 disk.dm-0.in‚0.078125 disk.sda1.out‚0 network.sent‚ 1.609077e+06) disk./.usagePercent‚75.76201643954515! disk.sda2.out‚147.98095703125 disk.rbd0.out‚ 0.279296875 proc.cpu.threads‚17 disk.nvme0n1.in‚ 0.02734375 disk.rbd0.in‚0 disk.nvme1n1.in‚ 0.05078125 disk.sdb1.out‚0 disk.rbd2.out‚ 0.29150390625$ proc.memory.availableMB‚1423.625# proc.memory.rssMB‚ 2893.44921875 disk.rbd2.in‚0 disk.md0p1.in‚ 0.0703125 disk.md0p1.out‚ 142.7421875* proc.memory.percent‚17.660212516784668 cpu‚25.050616657781923 disk.sdb.out‚147.98095703125 disk.nvme1n1.out‚ 5469.5625 disk.dm-0.out‚ 12825.9140625% disk./.usageGB‚2709.6166801452637! disk.nvme0n1.out‚ 7356.3515625 disk.sda.in‚ 0.0390625 disk.md0.in‚ 0.0703125 disk.sdb.in‚0.03125 disk.sda2.in‚ 0.0390625$ memory_percent‚91.31088256835938’ÏÓ’ò‚(:‡ ΋êÔçë»ÿ gpu.0.gpu‚47.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚168.992' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.248 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ƒýðÀ_ó‚(:Ô ΋êÔÿ’õÿ disk.rbd0.out‚ 0.279296875" disk.nvme0n1.out‚ 7356.51953125! disk.sdb2.out‚149.72509765625 disk.sda1.out‚0 disk.nvme0n1.in‚ 0.02734375 disk.sdb2.in‚0.03125 disk.dm-0.in‚0.078125 disk.md0p1.out‚ 144.4296875 disk.sdb.out‚149.72509765625 disk.sdb1.out‚0) disk./.usagePercent‚75.76549677270829 disk.dm-0.out‚ 12999.640625 disk.sda.out‚149.72509765625 disk.md0.out‚ 144.4296875( proc.memory.availableMB‚ 1423.2578125# proc.memory.rssMB‚ 2893.44921875" disk.nvme1n1.out‚ 5643.12109375 disk.md0.in‚ 0.0703125 network.sent‚ 1.615345e+06$ disk./.usageGB‚2709.741153717041 disk.sda.in‚ 0.0390625 disk.sda2.in‚ 0.0390625 disk.rbd2.in‚0* proc.memory.percent‚17.660212516784668 disk.sdb.in‚0.03125! disk.sda2.out‚149.72509765625 disk.rbd0.in‚0 disk.nvme1n1.in‚ 0.05078125 disk.rbd2.out‚ 0.29150390625$ memory_percent‚91.31312370300293 proc.cpu.threads‚17 network.recv‚490421 disk.sda1.in‚0 disk.md0p1.in‚ 0.0703125 disk.sdb1.in‚0 cpu‚25.050419405107696¤êÈ_”ô‚(:‰ Ý‹êÔðï› gpu.0.gpu‚28.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚152.899' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚38.22475 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0èÃ0Kõ‚(:À Ý‹êÔ͈„ disk.rbd0.out‚ 0.279296875 disk.nvme1n1.in‚ 0.05078125 disk.sdb2.in‚0.03125 disk.sda1.out‚0 disk.dm-0.in‚0.078125 disk.nvme0n1.in‚ 0.02734375# proc.memory.rssMB‚ 2893.44921875 network.sent‚ 1.622929e+06 disk.sdb.in‚0.03125 disk.sda.out‚ 151.3359375 disk.rbd0.in‚0 disk.md0.out‚ 145.984375 disk.sdb2.out‚ 151.3359375 disk.md0p1.in‚ 0.0703125 disk.md0p1.out‚ 145.984375# memory_percent‚91.3123369216919* proc.memory.percent‚17.660212516784668 cpu‚25.049704031054237 proc.cpu.threads‚17 disk.sdb1.in‚0 disk.rbd2.out‚ 0.29150390625 disk.sda1.in‚0 disk.dm-0.out‚ 13086.453125 disk.sdb.out‚ 151.3359375! disk.nvme1n1.out‚ 5729.8671875 network.recv‚493912) disk./.usagePercent‚75.76899054509846! disk.nvme0n1.out‚ 7356.5859375 disk.sda.in‚ 0.0390625 disk.sda2.out‚ 151.3359375 disk.md0.in‚ 0.0703125) proc.memory.availableMB‚ 1423.38671875 disk.rbd2.in‚0 disk.sdb1.out‚0$ disk./.usageGB‚2709.866107940674 disk.sda2.in‚ 0.0390625ùÑÚ¨2ö- _wandbruntime‚1339  _runtime‚1339ÎÖésÄ÷¾ % lr-AdamW/pg1‚0.000850606302635358 % lr-AdamW/pg2‚0.000850606302635358 trainer/global_step‚1199 % _timestamp‚1.7885117143762496e+09 _runtime‚1339.846638721 _step‚92†‘ÁVÈø % lr-AdamW/pg1‚0.000850606302635358 % lr-AdamW/pg2‚0.000850606302635358 trainer/global_step‚1199 % _timestamp‚1.7885117143762496e+09 _runtime‚1339.846638721 _step‚92\}W2ù- _wandbruntime‚1339  _runtime‚1339œÃ³ãúÙO KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚30188 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚108596813107200 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1331258750696 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 246679040 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1174999693 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1336250762752 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚29580093 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚96917110625280 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚96917110625280 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚107203364149248 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚95581776506368 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24590 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 229626880 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚93597673039120 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1330361346824 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1176280064 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 trainer/global_step‚1199 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚23888254 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30188 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1336250762752 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚95581758503424 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1335352121856 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 253566976 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 % _timestamp‚1.7885117143773534e+09 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13954824192 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚29604683 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚96918027269120 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13954824192 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 398533632 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1176280064 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 898640896 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1191 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚108596559540224 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 916643840 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚95581758503424 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30433 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 A7DeviceStatsMonitor.on_¸H{ütrain_batch_start/active.all.peak‚30433 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 898640896 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1335352121856 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚96918027269120 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚21766185 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1393425017856 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚94928034385944 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24592 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚23888256 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚107203388089344 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚95581776506368 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚21766825 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚94928948829176 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510207117 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚53468347 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚14857253 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914443232 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 916643840 _runtime‚1339.848005878 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 916643840 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚95581758503424 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚30433 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚30433 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 898640896 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1335352121856 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚96918027269120 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚21766185 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1393425017856 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚94928034385944 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚24592 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚23888256 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚107203388089344 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚95581776506368 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚21766825 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚94928948829176 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510207117 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚53468347 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚14857253 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914443232 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 916643840 _runtime‚1339.848005878 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚96917110625280 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚96917110625280 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚107203364149248 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚95581776506368 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚24590 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 229626880 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚93597673039120 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1330361346824 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1176280064 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 trainer/global_step‚1199 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚23888254 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚30188 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1336250762752 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚95581758503424 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1335352121856 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 253566976 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 % _timestamp‚1.7885117143773534e+09 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13954824192 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚29604683 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚96918027269120 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13954824192 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 398533632 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1176280064 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 898640896 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1191 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚108596559540224 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 IDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510207117 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24618 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13954824192 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 898654208 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚21784328 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚96998770381312 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚95008037584830 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6915322 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚95007123141390 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚53512836 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1176280064 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914443440 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚95661388798976 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1394380043264 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚14869629 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1336464925184 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1332367412222 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897404080 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚625 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 238873600 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 398533632 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1337363579392 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13954824192 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24616 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚53537454 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚21784953 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 246679040 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30188 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚107292683369984 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚23908152 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚95661406801920 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚95661406801920 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚623 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚14869631 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 _runtime‚ 1340.81632273 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 916657152 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚93675670172608 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚23908154 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24616 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚95661388798976 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1336464925184 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1337363579392 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚1199 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚29629300 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6914699 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚108687063413248 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚29629300 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚53512836 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30433 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚23908152 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚108687302286848 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚93675653133248*LäþæM N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13954824192 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 898654208 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚21784328 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚96998770381312 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚95008037584830 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6915322 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚95007123141390 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚53512836 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1176280064 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914443440 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚95661388798976 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1394380043264 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚14869629 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1336464925184 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1332367412222 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897404080 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚625 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 238873600 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 398533632 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1337363579392 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13954824192 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚24616 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚53537454 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚21784953 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 246679040 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚30188 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚107292683369984 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚23908152 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚95661406801920 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚95661406801920 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚623 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚14869631 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 _runtime‚ 1340.81632273 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 916657152 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚93675670172608 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚23908154 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚24616 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚95661388798976 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1336464925184 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1337363579392 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚1199 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚29629300 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6914699 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚108687063413248 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚29629300 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚53512836 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚30433 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚23908152 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚108687302286848 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚93675653133248 R?Deviceû«oFStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚96997853724160 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚29604684 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1331470008142 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1174999693 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚23908154 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1166 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 % _timestamp‚1.7885117153455207e+09 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚96998770381312 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510207117 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚24618^;Ö£!2ÿ- _wandbruntime‚1340  _runtime‚1340S”BüR€Ì 0loss/FRt4/assignment_loss‚3.9970361505235945 2loss/SRqqt4/assignment_loss‚2.4011369390147075 /loss/FRt4/detection_loss‚3.2066544010525657 1loss/SRqqt4/detection_loss‚2.4972240839685713 /loss/FBt2/assignment_loss‚0.266632367817304 1loss/SRqqt2/detection_loss‚1.1391295814706432 1loss/SRqqt2/assignment_loss‚1.131048064078054 /loss/FBt1/detection_loss‚1.8874554375807444 epoch‚0 $ _timestamp‚1.788511715347673e+09 /loss/FRt1/assignment_loss‚4.946304896298577 .loss/FBt3/detection_loss‚5.286149303118388 0loss/FBt1/assignment_loss‚0.8048404633998871 0loss/SRqqt3/detection_loss‚1.689035500048614 /loss/FRt2/detection_loss‚1.8033226600715093 0loss/FRt2/assignment_loss‚2.0218990881528174 0loss/FRt3/assignment_loss‚3.9933018883069358 1loss/SRqqt3/assignment_loss‚1.852238198101032 $loss/total_loss‚53.2508375358825 _step‚95 /loss/SRqqt1/detection_loss‚2.44893791526556 _runtime‚1340.817981458 .loss/FBt4/detection_loss‚3.747761300631932 trainer/global_step‚1199 /loss/FRt1/detection_loss‚2.2849117429817425 1loss/SRqqt1/assignment_loss‚2.383093474432826 1loss/FBt3/assignment_loss‚0.28397709073442456 0loss/FBt4/assignment_loss‚0.3629504314490727 .loss/FRt3/detection_loss‚2.190745030255879 /loss/FBt2/detection_loss‚0.6250515271271213_MVÐ epoch‚0 $ _timestamp‚1.788511715347673e+09 /loss/FRt1/assignment_loss‚4.946304896298577 .loss/FBt3/detection_loss‚5.286149303118388 0loss/FBt1/assignment_loss‚0.8048404633998871 0loss/SRqqt3/detection_loss‚1.689035500048614 /loss/FRt2/detection_loss‚1.8033226600715093 0loss/FRt2/assignment_loss‚2.0218990881528174 0loss/FRt3/assignment_loss‚3.9933018883069358 1loss/SRqqt3/assignment_loss‚1.852238198101032 $loss/total_loss‚53.2508375358825 _step‚95 /loss/SRqqt1/detection_loss‚2.44893791526556 _runtime‚1340.817981458 .loss/FBt4/detection_loss‚3.747761300631932 trainer/global_step‚1199 /loss/FRt1/detection_loss‚2.2849117429817425 1loss/SRqqt1/assignment_loss‚2.383093474432826 1loss/FBt3/assignment_loss‚0.28397709073442456 0loss/FBt4/assignment_loss‚0.3629504314490727 .loss/FRt3/detection_loss‚2.190745030255879 /loss/FBt2/detection_loss‚0.6250515271271213 0loss/FRt4/assignment_loss‚3.9970361505235945 2loss/SRqqt4/assignment_loss‚2.4011369390147075 /loss/FRt4/detection_loss‚3.2066544010525657 1loss/SRqqt4/detection_loss‚2.4972240839685713 /loss/FBt2/assignment_loss‚0.266632367817304 1loss/SRqqt2/detection_loss‚1.1391295814706432 1loss/SRqqt2/assignment_loss‚1.131048064078054 /loss/FBt1/detection_loss‚1.8874554375807444_øÀHÊ“‚‚(:ˆ ì‹êÔ«Œþ gpu.0.gpu‚2.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚37.0 gpu.0.powerWatts‚114.131' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.53275 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0BÀO±]ƒ‚(:Ò ì‹êÔŽÓæþ disk.rbd0.in‚0 disk.rbd0.out‚ 0.279296875 disk.rbd2.in‚0 disk.rbd2.out‚ 0.29150390625 disk.dm-0.in‚0.078125 disk.dm-0.out‚ 13174.984375 disk.md0.out‚ 147.84765625 disk.sda1.in‚0 disk.sda1.out‚0 disk.sdb.out‚153.2705078125 disk.sda.in‚ 0.0390625) proc.memory.percent‚17.66045093536377 network.sent‚ 1.626728e+06 disk.sdb2.out‚153.2705078125 disk.sda.out‚153.2705078125$ memory_percent‚91.31486415863037$ disk./.usageGB‚2710.029613494873" disk.nvme0n1.out‚ 7357.03515625# proc.memory.rssMB‚ 2893.48828125 cpu‚25.049180295791828) disk./.usagePercent‚75.77356222882626 disk.md0p1.in‚ 0.0703125 disk.nvme0n1.in‚ 0.02734375 disk.sdb.in‚0.03125 disk.sda2.in‚ 0.0390625 disk.nvme1n1.in‚ 0.05078125 proc.cpu.threads‚17 network.recv‚498250 disk.sdb2.in‚0.03125 disk.md0p1.out‚ 147.84765625" disk.nvme1n1.out‚ 5817.94921875 disk.md0.in‚ 0.0703125 disk.sda2.out‚153.2705078125 disk.sdb1.in‚0 disk.sdb1.out‚0) proc.memory.availableMB‚ 1422.97265625T-aà”„‚(:‰ û‹êÔÙÐÿ gpu.0.gpu‚49.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚36.0 gpu.0.powerWatts‚114.131' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.53275 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0£šR½W…‚(:Ì û‹êÔõ®š€ disk.nvme0n1.in‚ 0.02734375 disk.sdb.in‚0.03125 disk.sdb.out‚154.80322265625% disk./.usageGB‚2710.1680793762207 disk.dm-0.in‚0.078125 disk.dm-0.out‚13332.50390625 disk.md0p1.out‚149.3125 disk.sda.in‚ 0.0390625 disk.sda2.in‚ 0.0390625 disk.sdb2.in‚0.03125 disk.sda.out‚154.80322265625 cpu‚25.087435029864878) disk./.usagePercent‚75.77743379282121 disk.rbd2.in‚0' proc.memory.availableMB‚ 1340.765625 disk.rbd0.in‚0 disk.nvme1n1.in‚ 0.05078125 disk.md0.in‚ 0.0703125 disk.sda1.in‚0$ memory_percent‚91.81661605834961! disk.sda2.out‚154.80322265625 disk.rbd0.out‚ 0.279296875 disk.sdb1.in‚0 disk.sda1.out‚0 network.recv‚502588 disk.md0.out‚149.3125 disk.rbd2.out‚ 0.29150390625 disk.md0p1.in‚ 0.0703125 proc.cpu.threads‚17" disk.nvme1n1.out‚ 5975.46484375! disk.nvme0n1.out‚ 7357.0390625 network.sent‚ 1.675651e+06 disk.sdb1.out‚0! disk.sdb2.out‚154.80322265625" proc.memory.rssMB‚ 2976.0390625( proc.memory.percent‚18.1643009185791iÍ€_†‚(:Ô ŠŒêÔ¯¿Ù disk.sda.in‚ 0.0390625 disk.sda2.in‚ 0.0390625 disk.md0.in‚ 0.0703125$ memory_percent‚91.30902290344238 cpu‚25.12753736559961 disk.rbd2.in‚0! disk.sdb2.out‚156.34130859375( proc.memory.availableMB‚ 1423.9296875" disk.nvme1n1.out‚ 6133.15234375! disk.sda2.out‚156.34130859375 disk.rbd2.out‚ 0.29150390625# proc.memory.rssMB‚ 2892.85546875* proc.memory.percent‚17.656588554382324 proc.cpu.threads‚17 disk.rbd0.out‚ 0.279296875 disk.md0.out‚ 150.77734375 disk.sda1.out‚0$ disk./.usageGB‚2710.171920776367 disk.sda1.in‚0 disk.md0p1.out‚ 150.77734375 disk.dm-0.in‚0.078125 disk.dm-0.out‚13490.24609375 disk.md0p1.in‚ 0.0703125 network.sent‚ 1.682019e+06) disk./.usagePercent‚75.77754119997707 disk.rbd0.in‚0 disk.nvme1n1.in‚ 0.05078125 disk.sdb2.in‚0.03125 disk.nvme0n1.in‚ 0.02734375 disk.sdb.out‚156.34130859375 network.recv‚507528 disk.sda.out‚156.34130859375 disk.nvme0n1.out‚ 7357.09375 disk.sdb.in‚0.03125 disk.sdb1.in‚0 disk.sdb1.out‚0±ZU4’‡‚(:‡ ŠŒêÔÙÇì gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚35.0 gpu.0.powerWatts‚113.466' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.3665 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ìÓg’ˆ‚(:‡ ™ŒêԷʯÿ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚35.0 gpu.0.powerWatts‚113.466' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.3665 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0|wOZ‰‚(:Ï ™ŒêÔáÀ…€ disk.sda1.in‚0# proc.memory.rssMB‚ 2892.97265625 proc.cpu.threads‚17 disk.dm-0.out‚13581.27734375 disk.sdb.out‚158.42529296875 disk.rbd0.out‚ 0.279296875% disk./.usageGB‚2710.2345809936523" disk.nvme1n1.out‚ 6224.11328125 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.rbd2.in‚0$ memory_percent‚91.31011962890625 network.recv‚510194 disk.md0.out‚ 152.7890625 disk.md0p1.in‚ 0.0703125 disk.md0p1.out‚ 152.7890625# proc.memory.availableMB‚1423.75) proc.memory.percent‚17.65730381011963 disk.dm-0.in‚0.078125! disk.sda2.out‚158.42529296875 network.sent‚ 1.689323e+06) disk./.usagePercent‚75.77929320587769 disk.nvme1n1.in‚ 0.05078125 disk.md0.in‚ 0.0703125 disk.rbd2.out‚ 0.35400390625! disk.nvme0n1.out‚ 7357.1640625 disk.sda.in‚ 0.0390625 disk.sdb2.in‚0.03125! disk.sdb2.out‚158.42529296875 disk.sda1.out‚0 disk.nvme0n1.in‚ 0.02734375 disk.sda.out‚158.42529296875 disk.sda2.in‚ 0.0390625 disk.sdb.in‚0.03125 disk.rbd0.in‚0 cpu‚25.16321308046007š³grŠbm JDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.current"trainer/global_step2:JPkN’©p‹bk HDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.freed"trainer/global_step2:JP‰·ëvŒbq NDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.current"trainer/global_step2:JPjËwbr ODeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.allocated"trainer/global_step2:JPÅ\£~dŽb_ DeviceStatsMonitor.on_validation_batch_start/segment.all.freed"trainer/global_step2:JP-ÜA9oÓbj GDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.current"trainer/global_step2:JPe$a5eÔb` =DeviceStatsMonitor.on_validation_batch_start/active.all.freed"trainer/global_step2:JP]wVlÕbg DDeviceStatsMonitor.on_validation_batch_start/active.large_pool.freed"trainer/global_step2:JP2ëZftÖbo LDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.freed"trainer/global_step2:JP+0'ëv×bq NDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.allocated"trainer/global_step2:JP6‰ÞpØbk HDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.current"trainer/global_step2:JPŠ i˜tÙbo LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.peak"trainer/global_step2:JPjTuÚbp MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.freed"trainer/global_step2:JP­¢”lÛbg DDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.peak"trainer/global_step2:JPŒÖn*nÜbi FDeviceStatsMonitor.on_validation_batch_start/active.large_pool.current"trainer/global_step2:JPT¡ClqÝbl IDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.peak"trainer/global_step2:JP·Ù*yÞbt QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.peak"trainer/global_step2:JP2Cá“jßbe BDeviceStatsMonitor.on_validation_batch_start/segment.all.allocated"trainer/global_step2:JPúhyÚpàbk HDeviceStatsMonitor.on_validation_batch_start/active.small_pool.allocated"trainer/global_step2:JPœ¥:Ÿpábk HDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.freed"trainer/global_step2:JPÃìíªsâbn KDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.peak"trainer/global_step2:JP„td³|ãbw TDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.current"trainer/global_step2:JPÂÞFzzäbu RDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.freed"trainer/global_step2:JPšª_iåbd ADeviceStatsMonitor.on_validation_batch_start/num_sync_all_streams"trainer/global_step2:JPhKk÷uæbp MDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.current"trainer/global_step2:JP¨P¡xçbs PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.allocated"trainer/global_step2:JP(r0yxèbs PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.allocated"trainer/global_step2:JP ÒrËlébg DDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.peak"trainer/global_step2:JP½R×mêbh EDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.freed"trainer/global_step2:JPcënëbi FDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.freed"trainer/global_step2:JP¿•k‘wìbr ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.current"trainer/global_step2:JPPèÈ·síbn KDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.peak"trainer/global_step2:JPþAÎ#fîba >DeviceStatsMonitor.on_validation_batch_start/num_alloc_retries"trainer/global_step2:JPEsîkïbf CDeviceStatsMonitor.on_validation_batch_start/oversize_segments.peak"trainer/global_step2:JPŠO7uðbp MDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.freed"trainer/global_step2:JPh¥°wñbr ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.current"trainer/global_step2:JP¯ ¾íxòbs PDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.allocated"trainer/global_step2:JP@Äõsóbn KDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.allocated"trainer/global_step2:JP ËÂÈsôbn KDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.freed"trainer/global_step2:JP:ð!wõbr ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.current"trainer/global_step2:JPg=I\oöbj GDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.current"trainer/global_step2:JPŒ³> k÷bf CDeviceStatsMonitor.on_validation_batch_start/active.large_pool.peak"trainer/global_step2:JP}‚H™møbh EDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.peak"trainer/global_step2:JPïi3mtùbo LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.peak"trainer/global_step2:JP6Ö[=húbc @DeviceStatsMonitor.on_validation_batch_start/allocation.all.peak"trainer/global_step2:JPíG÷™mûbh EDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.freed"trainer/global_step2:JP´l›ˆnübi FDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.freed"trainer/global_step2:JP ôsýbn KDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.peak"trainer/global_step2:JP³¤\ôtþbo LDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.current"trainer/global_step2:JPÁäY@gÿbb ?DeviceStatsMonitor.on_validation_batch_start/num_oom_rejections"trainer/global_step2:JPR03¾n€bi FDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.peak"trainer/global_step2:JP¡ãEOtbo LDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.current"trainer/global_step2:JP3†­›o‚bj GDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.peak"trainer/global_step2:JP”¶]ˆtƒbo LDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.freed"trainer/global_step2:JP¯!¡¾e„b` =DeviceStatsMonitor.on_validation_batch_start/num_device_alloc"trainer/global_step2:JP!ŸXÔ2…- _wandbruntime‚1409  _runtime‚1409p;ùva†ÄT YLDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.allocated‚24835760 cQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.allocated‚ 1366220787988 ^JDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.allocated‚100374405315072 \PDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.allocated‚7233133 NHDeviceStatsMonitor.on_validation_batch_start/oversize_segments.allocated‚0 \LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.peak‚ 12392107008 MCDeviceStatsMonitor.on_validation_batch_start/allocation.all.current‚19735 I@DeviceStatsMonitor.on_validation_batch_start/segment.all.current‚1171 QIDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.allocated‚607 I?DeviceStatsMonitor.on_validation_batch_start/active.all.current‚19735 XIDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.peak‚ 1176280064 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 LFDeviceStatsMonitor.on_validation_batch_start/oversize_segments.current‚0 [LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.peak‚ 1174999693 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.freed‚0 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.freed‚15443573 dRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.freed‚ 1429026645504 YIDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.allocated‚ 14220787712 \ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.current‚18002944 ]JDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.allocated‚98182446209124 VHDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.current‚ 690180844 UEDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.peak‚ 13510207117 ^NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.current‚ 13037993984 dQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.allocated‚99003071974912 VIDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.allocated‚22676708 XJDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.peak‚ 398533632 RBDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.peak‚ 13954824192 bTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.current‚ 145272320 ;5DeviceStatsMonitor.on_validation_batch_start/num_ooms‚0 OGDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.current‚607 LDDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.peak‚564 _step‚96 NADeviceStatsMonitor.on_validation_batch_start/active.all.allocated‚55116258 MCDeviceStatsMonitor.on_validation_batch_start/active.small_pool.peak‚30188 ZKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 REDeviceStatsMonitor.on_validation_batch_start/allocation.all.allocated‚55116258 YLDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.allocated‚30280498 QGDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.peak‚30188 hVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.allocated‚ 1429171917824 QIDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.allocated‚564 % _timestamp‚1.7885117845251362e+09 PFDeviceStatsMonitor.on_validation_batch_start/active.small_pool.current‚19733 SEDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.current‚ 692717056 \JDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.freed‚ 1370658626048 NADeviceStatsMonitor.on_validation_batch_start/allocation.all.freed‚55096523 jVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.allocated‚110813604424704 TDDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.peak‚ 14220787712 D>DeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 OGDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.current‚202 J=DeviceStatsMonitor.on_validation_batch_start/active.all.freed‚55096523 QDDeviceStatsMonitor.on_validation_batch_start/active.large_pool.freed‚24835758 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.freed‚0 `NDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.allocated‚ 1371333340160 VHDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.current‚ 692717056 \LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 _MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.freed‚ 1370658626048 MDDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.peak‚1191 _runtime‚1409.996261256 LFDeviceStatsMonitor.on_validation_batch_start/active.large_pool.current‚2 YIDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.peak‚ 12835443712 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 KBDeviceStatsMonitor.on_validation_batch_start/segment.all.allocated‚1171 UHDeviceStatsMonitor.on_validation_batch_start/active.small_pool.allocated‚30280498 UHDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.freed‚24835758 RKDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.peak‚35 aTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.current‚23940096 fRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.freed‚110813580484608 GADeviceStatsMonitor.on_validation_batch_start/num_sync_all_streams‚2 [MDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.current‚ 169212416 `PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 _PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 LDDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.peak‚607 REDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.freed‚22676506 ZFDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.freed‚100373712598016 ]ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.current‚ 674714112 TKDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.peak‚1166 D>DeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_validation_batch_start/oversize_segments.peak‚0 `MDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.freed‚96816208381776 ]ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.current‚ 673141484 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.allocated‚15443575 QKDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.allocated‚0 _KDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.freed‚112242607130112 \ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.lƒ"àé arge_pool.current‚17039360 WGDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.current‚ 14220787712 LCDeviceStatsMonitor.on_validation_batch_start/active.large_pool.peak‚3677 UEDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.peak‚ 13954824192 [LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.peak‚ 1176280064 J@DeviceStatsMonitor.on_validation_batch_start/allocation.all.peak‚30433 KEDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.freed‚0 YFDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.freed‚98181756028280 [KDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 YLDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.current‚18002944 E?DeviceStatsMonitor.on_validation_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.peak‚0 ZLDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.current‚ 674714112 PGDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.peak‚3677 XLDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.freed‚7232933 F=DeviceStatsMonitor.on_validation_batch_start/num_device_alloc‚1171 PJDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.current‚2 UHDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.freed‚30260765 VNDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.current‚200 cODeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.allocated‚112242776342528 BDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 OGDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.current‚202 J=DeviceStatsMonitor.on_validation_batch_start/active.all.freed‚55096523 QDDeviceStatsMonitor.on_validation_batch_start/active.large_pool.freed‚24835758 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.freed‚0 `NDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.allocated‚ 1371333340160 VHDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.current‚ 692717056 \LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 _MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.freed‚ 1370658626048 MDDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.peak‚1191 _runtime‚1409.996261256 LFDeviceStatsMonitor.on_validation_batch_start/active.large_pool.current‚2 YIDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.peak‚ 12835443712 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 KBDeviceStatsMonitor.on_validation_batch_start/segment.all.allocated‚1171 UHDeviceStatsMonitor.on_validation_batch_start/active.small_pool.allocated‚30280498 UHDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.freed‚24835758 RKDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.peak‚35 aTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.current‚23940096 fRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.freed‚110813580484608 GADeviceStatsMonitor.on_validation_batch_start/num_sync_all_streams‚2 [MDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.current‚ 169212416 `PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 _PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 LDDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.peak‚607 REDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.freed‚22676506 ZFDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.freed‚100373712598016 ]ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.current‚ 674714112 TKDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.peak‚1166 D>DeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_validation_batch_start/oversize_segments.peak‚0 `MDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.freed‚96816208381776 ]ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.current‚ 673141484 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.allocated‚15443575 QKDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.allocated‚0 _KDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.freed‚112242607130112 \ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.current‚ 14220787712 LCDeviceStatsMonitor.on_validation_batch_start/active.large_pool.peak‚3677 UEDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.peak‚ 13954824192 [LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.peak‚ 1176280064 J@DeviceStatsMonitor.on_validation_batch_start/allocation.all.peak‚30433 KEDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.freed‚0 YFDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.freed‚98181756028280 [KDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 YLDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.current‚18002944 E?DeviceStatsMonitor.on_validation_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.peak‚0 ZLDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.current‚ 674714112 PGDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.peak‚3677 XLDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.freed‚7232933 F=DeviceStatsMonitor.on_validation_batch_start/num_device_alloc‚1171 PJDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.current‚2 UHDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.freed‚30260765 VNDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.current‚200 cODeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.allocated‚112242776342528 B°Aw³br ODeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.allocated"trainer/global_step2:JP3nr´bm JDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.peak"trainer/global_step2:JP.aãárµbm JDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.allocated"trainer/global_step2:JPdƒ:ýq¶bl IDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.freed"trainer/global_step2:JP‡©ëx·bs PDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.freed"trainer/global_step2:JP Îñ1p¸bk HDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.current"trainer/global_step2:JPrM1Zq¹bl IDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.allocated"trainer/global_step2:JPƒâAnºbi FDeviceStatsMonitor.on_validation_batch_end/active.small_pool.allocated"trainer/global_step2:JP¸<Éi»bd ADeviceStatsMonitor.on_validation_batch_end/active.small_pool.peak"trainer/global_step2:JPÑlàÇw¼br ODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.allocated"trainer/global_step2:JP:D+å|½bw TDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.allocated"trainer/global_step2:JPëÉ*Õj¾be BDeviceStatsMonitor.on_validation_batch_end/active.small_pool.freed"trainer/global_step2:JPºi¿bd ADeviceStatsMonitor.on_validation_batch_end/active.large_pool.peak"trainer/global_step2:JP ™!mÀbh EDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.allocated"trainer/global_step2:JP,Œ\rÁbm JDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.freed"trainer/global_step2:JPžoÙ^wÂbr ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.peak"trainer/global_step2:JPlÉÁlÃbg DDeviceStatsMonitor.on_validation_batch_end/active.small_pool.current"trainer/global_step2:JPÀè-jÄbe BDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.peak"trainer/global_step2:JPéãøZwÅbr ODeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.allocated"trainer/global_step2:JPi\ÓpcÆb^ ;DeviceStatsMonitor.on_validation_batch_end/segment.all.peak"trainer/global_step2:JP7÷$rÇbm JDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.current"trainer/global_step2:JP}3<*rÈbm JDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.allocated"trainer/global_step2:JPSHEH|Ébw TDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.allocated"trainer/global_step2:JPIkökÊbf CDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.freed"trainer/global_step2:JP¨‹oËbj GDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.peak"trainer/global_step2:JPãgeÌb` =DeviceStatsMonitor.on_validation_batch_end/active.all.current"trainer/global_step2:JP›Ý´1lÍbg DDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.freed"trainer/global_step2:JP¸}itÎbo LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.current"trainer/global_step2:JP±…||iÏbd ADeviceStatsMonitor.on_validation_batch_end/allocation.all.current"trainer/global_step2:JP&_'QwÐbr ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.peak"trainer/global_step2:JPOæÿmÑbh EDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.freed"trainer/global_step2:JP³aÖ'mÒbh EDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.current"trainer/global_step2:JPÆî0hÓbc @DeviceStatsMonitor.on_validation_batch_end/active_bytes.all.peak"trainer/global_step2:JP…rKªsÔbn KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.freed"trainer/global_step2:JPö‰lÞqÕbl IDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.peak"trainer/global_step2:JPc(RlÖbg DDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.peak"trainer/global_step2:JP0 ì's×bn KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.freed"trainer/global_step2:JP¹ßV—oØbj GDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.allocated"trainer/global_step2:JP¾<ÉtÙbo LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.current"trainer/global_step2:JP€|!MwÚbr ODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.allocated"trainer/global_step2:JPÅ+VÔqÛbl IDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.peak"trainer/global_step2:JP×>FqlÜbg DDeviceStatsMonitor.on_validation_batch_end/oversize_segments.current"trainer/global_step2:JPgÖ¾®vÝbq NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.allocated"trainer/global_step2:JPÀ¸’4hÞbc @DeviceStatsMonitor.on_validation_batch_end/segment.all.allocated"trainer/global_step2:JPÁI®¶kßbf CDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.freed"trainer/global_step2:JP…`"9kàbf CDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.peak"trainer/global_step2:JPŽ ÞDeviceStatsMonitor.on_validation_batch_end/segment.all.current"trainer/global_step2:JPù2 jäbe BDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.peak"trainer/global_step2:JPøòWråbm JDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.current"trainer/global_step2:JP ç‹uæbp MDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.current"trainer/global_step2:JPÕ,i>mçbh EDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.peak"trainer/global_step2:JPëESpèbk HDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.peak"trainer/global_step2:JPœ=yñnébi FDeviceStatsMonitor.on_validation_batch_end/active.large_pool.allocated"trainer/global_step2:JPïÿ°atêbo LDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.allocated"trainer/global_step2:JPC_ÏÙpëbk HDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.freed"trainer/global_step2:JP±JgDrìbm JDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.freed"trainer/global_step2:JP¡›Ígíbb ?DeviceStatsMonitor.on_validation_batch_end/num_sync_all_streams"trainer/global_step2:JPà•»\sîbn KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.freed"trainer/global_step2:JP2¼”nïbi FDeviceStatsMonitor.on_validation_batch_end/oversize_segments.allocated"trainer/global_step2:JP¦TgBjðbe BDeviceStatsMonitor.on_validation_batch_end/oversize_segments.freed"trainer/global_step2:JP—F¿'pñbk HDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.allocated"trainer/global_step2:JP¸ë"2oòbj GDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.allocated"trainer/global_step2:JP ™}ÿjóbe BDeviceStatsMonitor.on_validation_batch_end/active.large_pool.freed"trainer/global_step2:JPÅ™ä.rôbm JDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.peak"trainer/global_step2:JPQã·evõbq NDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.allocated"trainer/global_step2:JP¦¯\¤uöbp MDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.allocated"trainer/global_step2:JP΋òëz÷bu RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.current"trainer/global_step2:JPeSÀuøbp MDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.current"trainer/global_step2:JP;ŠÑmùbh EDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.current"trainer/global_step2:JP†By¢qúbl IDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.peak"trainer/global_step2:JPO‘j„pûbk HDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.allocated"trainer/global_step2:JPŽ© {oübj GDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.peak"trainer/global_step2:JPq¯+ûjýbe BDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.peak"trainer/global_step2:JPK€ñÂkþbf CDeviceStatsMonitor.on_validation_batch_end/allocation.all.allocated"trainer/global_step2:JP) _>fÿba >DeviceStatsMonitor.on_validation_batch_end/allocation.all.peak"trainer/global_step2:JPGQn€bi FDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.freed"trainer/global_step2:JPÑQ,%pbk HDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.current"trainer/global_step2:JP°Í&Un‚bi FDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.current"trainer/global_step2:JPJå}Lrƒbm JDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.freed"trainer/global_step2:JP­Ñx k„bf CDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.freed"trainer/global_step2:JPË•rG2…- _wandbruntime‚1411  _runtime‚1411÷  †ÎR KEDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.freed‚0 MEDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.current‚564 P@DeviceStatsMonitor.on_validation_batch_end/active_bytes.all.peak‚ 13954824192 ]KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.freed‚ 1371147188736 PIDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.peak‚35 JDDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.peak‚0 ^KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.freed‚96865978471600 trainer/global_step‚1208 WGDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.allocated‚ 14220787712 \LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.current‚ 13037993984 aODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.allocated‚ 1371821902848 RIDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.peak‚1166 JDDeviceStatsMonitor.on_validation_batch_end/oversize_segments.current‚0 ]NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 I@DeviceStatsMonitor.on_validation_batch_end/segment.all.allocated‚1171 ICDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.freed‚0 SCDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.peak‚ 13954824192 OEDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.peak‚30188 ZJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.peak‚ 12392107008 G>DeviceStatsMonitor.on_validation_batch_end/segment.all.current‚1171 RBDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.peak‚ 14220787712 WJDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.current‚18002944 ZMDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.current‚18002944 NEDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.peak‚3677 VHDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.peak‚ 398533632 % _timestamp‚1.7885117857777412e+09 SFDeviceStatsMonitor.on_validation_batch_end/active.large_pool.allocated‚24851036 _LDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.allocated‚99056643250688 ZHDeviceStatsMo¶8(L nitor.on_validation_batch_end/active_bytes.small_pool.freed‚ 1371147188736 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.freed‚15453013 E?DeviceStatsMonitor.on_validation_batch_end/num_sync_all_streams‚2 ]KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.freed‚ 1366034573228 LFDeviceStatsMonitor.on_validation_batch_end/oversize_segments.allocated‚0 HBDeviceStatsMonitor.on_validation_batch_end/oversize_segments.freed‚0 [HDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.allocated‚98232703225672 OGDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.allocated‚564 OBDeviceStatsMonitor.on_validation_batch_end/active.large_pool.freed‚24851034 ZJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.allocated‚15453015 aMDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.allocated‚112298714377216 `RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.current‚ 145272320 [MDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.current‚ 673141484 UEDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.current‚ 14220787712 YIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 \HDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.allocated‚100428465153536 WGDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.peak‚ 12835443712 JBDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.peak‚564 PCDeviceStatsMonitor.on_validation_batch_end/allocation.all.allocated‚55140308 H>DeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚30433 SFDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.freed‚24851034 RHDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.current‚19733 TFDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.current‚ 690180844 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.freed‚0 ICDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.freed‚0 [MDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.current‚ 674714112 @9DeviceStatsMonitor.on_validation_batch_end/max_split_size‚-1 ZMDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.current‚17039360 ^NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 JBDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.peak‚607 _step‚97 QCDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.current‚ 692717056 MEDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.current‚607 ^LDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.allocated‚ 1371821902848 YKDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.current‚ 169212416 @:DeviceStatsMonitor.on_validation_batch_end/num_device_free‚0 RLDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.current‚2 TFDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.current‚ 692717056 MEDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.current‚203 L?DeviceStatsMonitor.on_validation_batch_end/active.all.allocated‚55140308 H;DeviceStatsMonitor.on_validation_batch_end/active.all.freed‚55120573 JDDeviceStatsMonitor.on_validation_batch_end/active.large_pool.current‚2 ^KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.freed‚99056625247744 TGDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.allocated‚22691558 D;DeviceStatsMonitor.on_validation_batch_end/num_device_alloc‚1171 MGDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.current‚0 SCDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.peak‚ 13510207117 UADeviceStatsMonitor.on_validation_batch_end/active_bytes.all.freed‚100427772436480 [HDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.freed‚99056625247744 PCDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.freed‚22691355 93DeviceStatsMonitor.on_validation_batch_end/num_ooms‚0 XIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 D:DeviceStatsMonitor.on_validation_batch_end/active.all.peak‚30433 XDDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.freed‚100427772436480 TLDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.current‚201 BDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚1171 RBDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.peak‚ 14220787712 WJDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.current‚18002944 ZMDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.current‚18002944 NEDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.peak‚3677 VHDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.peak‚ 398533632 % _timestamp‚1.7885117857777412e+09 SFDeviceStatsMonitor.on_validation_batch_end/active.large_pool.allocated‚24851036 _LDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.allocated‚99056643250688 ZHDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.freed‚ 1371147188736 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.freed‚15453013 E?DeviceStatsMonitor.on_validation_batch_end/num_sync_all_streams‚2 ]KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.freed‚ 1366034573228 LFDeviceStatsMonitor.on_validation_batch_end/oversize_segments.allocated‚0 HBDeviceStatsMonitor.on_validation_batch_end/oversize_segments.freed‚0 [HDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.allocated‚98232703225672 OGDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.allocated‚564 OBDeviceStatsMonitor.on_validation_batch_end/active.large_pool.freed‚24851034 ZJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.allocated‚15453015 aMDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.allocated‚112298714377216 `RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.current‚ 145272320 [MDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.current‚ 673141484 UEDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.current‚ 14220787712 YIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 \HDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.allocated‚100428465153536 WGDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.peak‚ 12835443712 JBDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.peak‚564 PCDeviceStatsMonitor.on_validation_batch_end/allocation.all.allocated‚55140308 H>DeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚30433 SFDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.freed‚24851034 RHDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.current‚19733 TFDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.current‚ 690180844 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.freed‚0 ICDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.freed‚0 [MDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.current‚ 674714112 @9DeviceStatsMonitor.on_validation_batch_end/max_split_size‚-1 ZMDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.current‚17039360 ^NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 JBDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.peak‚607 _step‚97 QCDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.current‚ 692717056 MEDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.current‚607 ^LDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.allocated‚ 1371821902848 YKDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.current‚ 169212416 @:DeviceStatsMonitor.on_validation_batch_end/num_device_free‚0 RLDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.current‚2 TFDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.current‚ 692717056 MEDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.current‚203 L?DeviceStatsMonitor.on_validation_batch_end/active.all.allocated‚55140308 H;DeviceStatsMonitor.on_validation_batch_end/active.all.freed‚55120573 JDDeviceStatsMonitor.on_validation_batch_end/active.large_pool.current‚2 ^KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.freed‚99056625247744 TGDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.allocated‚22691558 D;DeviceStatsMonitor.on_validation_batch_end/num_device_alloc‚1171 MGDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.current‚0 SCDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.peak‚ 13510207117 UADeviceStatsMonitor.on_validation_batch_end/active_bytes.all.freed‚100427772436480 [HDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.freed‚99056625247744 PCDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.freed‚22691355 93DeviceStatsMonitor.on_validation_batch_end/num_ooms‚0 XIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 D:DeviceStatsMonitor.on_validation_batch_end/active.all.peak‚30433 XDDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.freed‚100427772436480 TLDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.current‚201 B­ÆJíbE "Purity/*FRt1SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPƒÚy7JîbE "Purity/*FRt*SRqqt1FBt/event_purity"trainer/global_step2:JP¼GÁ@ïb; particle/accuracy_2_of_5"trainer/global_step2:JPÛus;ðb6 jet/accuracy_7_of_7"trainer/global_step2:JP¡ òQJñbE "Purity/4FRt1SRqqt*FBt/event_purity"trainer/global_step2:JPI–i NòbI &Purity/3FRt1SRqqt2FBt/event_proportion"trainer/global_step2:JPêêRNóbI &Purity/4FRt2SRqqt2FBt/event_proportion"trainer/global_step2:JPa°y÷@ôb; particle/accuracy_2_of_4"trainer/global_step2:JP°+`úJõbE "Purity/3FRt2SRqqt1FBt/SRqqt_purity"trainer/global_step2:JP’FŒ~HöbC Purity/4FRt0SRqqt0FBt/FRt_purity"trainer/global_step2:JP?Y¿H÷bC Purity/*FRt1SRqqt*FBt/FBt_purity"trainer/global_step2:JPoQj‰JøbE "Purity/*FRt1SRqqt2FBt/event_purity"trainer/global_step2:JP£Ÿ–žNùbI &Purity/*FRt1SRqqt3FBt/event_proportion"trainer/global_step2:JPªDv‚NúbI &Purity/*FRt2SRqqt2FBt/event_proportion"trainer/global_step2:JP­0 "NûbI &Purity/1FRt2SRqqt*FBt/event_proportion"trainer/global_step2:JPDDy—JübE "Purity/2FRt2SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP QÞINýbI &Purity/4FRt0SRqqt4FBt/event_proportion"trainer/global_step2:JPÚ,¨SNþbI &Purity/4FRt3SRqqt1FBt/event_proportion"trainer/global_step2:JPHìþHÿbC Purity/*FRt*SRqqt0FBt/FRt_purity"trainer/global_step2:JP—fËhH€ bC Purity/1FRt*SRqqt3FBt/FBt_purity"trainer/global_step2:JP­TøµH bC Purity/1FRt0SRqqt3FBt/FRt_purity"trainer/global_step2:JP‡(#"J‚ bE "Purity/2FRt*SRqqt2FBt/SRqqt_purity"trainer/global_step2:JP0¥A°Hƒ bC Purity/1FRt3SRqqt1FBt/FBt_purity"trainer/global_step2:JP2/PJ„ bE "Purity/0FRt3SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP’ÜØ£H… bC Purity/4FRt*SRqqt*FBt/FRt_purity"trainer/global_step2:JP§}hJ† bE "Purity/0FRt2SRqqt3FBt/event_purity"trainer/global_step2:JPü8¡N‡ bI &Purity/1FRt*SRqqt4FBt/event_proportion"trainer/global_step2:JPŒŠ¢ƒNˆ bI &Purity/1FRt0SRqqt0FBt/event_proportion"trainer/global_step2:JP7¾µFN‰ bI &Purity/2FRt2SRqqt0FBt/event_proportion"trainer/global_step2:JPóêÝHŠ bC Purity/3FRt1SRqqt2FBt/FBt_purity"trainer/global_step2:JP_nSJ‹ bE "Purity/*FRt*SRqqt4FBt/SRqqt_purity"trainer/global_step2:JP3´ÐˆJŒ bE "Purity/0FRt2SRqqt3FBt/SRqqt_purity"trainer/global_step2:JPªöpJ® bE "Purity/1FRt3SRqqt1FBt/event_purity"trainer/global_step2:JPÐdÙH¯ bC Purity/3FRt1SRqqt1FBt/FBt_purity"trainer/global_step2:JPBUcµ;° b6 jet/accuracy_5_of_6"trainer/global_step2:JPnÞÃH± bC Purity/3FRt2SRqqt1FBt/FBt_purity"trainer/global_step2:JP>l§zH² bC Purity/*FRt*SRqqt*FBt/FBt_purity"trainer/global_step2:JP•OFiN³ bI &Purity/*FRt4SRqqt2FBt/event_proportion"trainer/global_step2:JPMµb÷N´ bI &Purity/2FRt0SRqqt*FBt/event_proportion"trainer/global_step2:JPÕ¦iRJµ bE "Purity/2FRt3SRqqt0FBt/event_purity"trainer/global_step2:JPŒ»b9N¶ bI &Purity/2FRt3SRqqt4FBt/event_proportion"trainer/global_step2:JP„‚¿ÂJ· bE "Purity/2FRt*SRqqt3FBt/event_purity"trainer/global_step2:JPÛQ16J¸ bE "Purity/1FRt3SRqqt2FBt/SRqqt_purity"trainer/global_step2:JPŸµƒvH¹ bC Purity/1FRt3SRqqt2FBt/FBt_purity"trainer/global_step2:JPÆW½rJº bE "Purity/*FRt0SRqqt1FBt/event_purity"trainer/global_step2:JPìg+áH» bC Purity/*FRt1SRqqt0FBt/FRt_purity"trainer/global_step2:JP@]QN¼ bI &Purity/1FRt0SRqqt3FBt/event_proportion"trainer/global_step2:JPñ­QN½ bI &Purity/2FRt1SRqqt0FBt/event_proportion"trainer/global_step2:JP#I‚´J¾ bE "Purity/3FRt1SRqqt*FBt/event_purity"trainer/global_step2:JPî1bN¿ bI &Purity/2FRt*SRqqt*FBt/event_proportion"trainer/global_step2:JPšÇŠHÀ bC Purity/3FRt2SRqqt*FBt/FRt_purity"trainer/global_step2:JP¼é?uNÁ bI &Purity/4FRt4SRqqt2FBt/event_proportion"trainer/global_step2:JPцfH bC Purity/3FRt1SRqqt*FBt/FRt_purity"trainer/global_step2:JPA“««Jà bE "Purity/3FRt1SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPpŒËHÄ bC Purity/*FRt3SRqqt*FBt/FRt_purity"trainer/global_step2:JP9—õNÅ bI &Purity/3FRt*SRqqt1FBt/event_proportion"trainer/global_step2:JP±ÜANÆ bI &Purity/3FRt1SRqqt*FBt/event_proportion"trainer/global_step2:JPsÂö`JÇ bE "Purity/3FRt2SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP Qê[JÈ bE "Purity/3FRt0SRqqt*FBt/event_purity"trainer/global_step2:JP ‘0JÉ bE "Purity/2FRt4SRqqt0FBt/event_purity"trainer/global_step2:JP¯Ä@A;Ê b6 jet/accuracy_1_of_7"trainer/global_step2:JP~jHË bC Purity/0FRt2SRqqt3FBt/FBt_purity"trainer/global_step2:JPõ.L-HÌ bC Purity/1FRt2SRqqt*FBt/FBt_purity"trainer/global_step2:JPªÚA8HÍ bC Purity/2FRt0SRqqt*FBt/FRt_purity"trainer/global_step2:JPD$«âJÎ bE "Purity/*FRt*SRqqt*FBt/event_purity"trainer/global_step2:JP­Ó¥NÏ bI &Purity/0FRt2SRqqt1FBt/event_proportion"trainer/global_step2:JPZÃyMNÐ bI &Purity/0FRt4SRqqt2FBt/event_proportion"trainer/global_step2:JPaD~NÑ bI &Purity/1FRt4SRqqt4FBt/event_proportion"trainer/global_step2:JP­À²JÒ bE "Purity/2FRt2SRqqt1FBt/event_purity"trainer/global_step2:JP•ð=HÓ bC Purity/2FRt4SRqqt0FBt/FRt_purity"trainer/global_step2:JP»p(NÔ bI &Purity/*FRt*SRqqt4FBt/event_proportion"trainer/global_step2:JPýhïNÕ bI &Purity/0FRt2SRqqt3FBt/event_proportion"trainer/global_step2:JPo¼æJÖ bE "Purity/1FRt*SRqqt2FBt/event_purity"trainer/global_step2:JP–„ÁJ× bE "Purity/1FRt0SRqqt0FBt/event_purity"trainer/global_step2:JP \©NØ bI &Purity/1FRt2SRqqt3FBt/event_proportion"trainer/global_step2:JP0$EJÙ bE "Purity/1FRt3SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPzi‹õNÚ bI &Purity/1FRt2SRqqt4FBt/event_proportion"trainer/global_step2:JPu¨A5NÛ bI &Purity/2FRt0SRqqt2FBt/event_proportion"trainer/global_step2:JPòÿDNÜ bI &Purity/2FRt2SRqqt*FBt/event_proportion"trainer/global_step2:JP7ËÍ£JÝ bE "Purity/3FRt1SRqqt0FBt/event_purity"trainer/global_step2:JP•JÞ bE "Purity/1FRt3SRqqt0FBt/event_purity"trainer/global_step2:JPf¦`Nß bI &Purity/3FRt0SRqqt0FBt/event_proportion"trainer/global_step2:JP.úkHà bC Purity/3FRt0SRqqt2FBt/FBt_purity"trainer/global_step2:JP1GÉHá bC Purity/2FRt0SRqqt1FBt/FBt_purity"trainer/global_step2:JPÒ*óNâ bI &Purity/2FRt4SRqqt3FBt/event_proportion"trainer/global_step2:JP:W~@ã b; particle/accuracy_1_of_3"trainer/global_step2:JPÛ£½@ä b; particle/accuracy_1_of_5"trainer/global_step2:JPî)õ„Hå bC Purity/*FRt*SRqqt4FBt/FBt_purity"trainer/global_step2:JPVÆ7@Næ bI &Purity/*FRt*SRqqt*FBt/event_proportion"trainer/global_step2:JP.xhNç bI &Purity/*FRt2SRqqt4FBt/event_proportion"trainer/global_step2:JPp Jè bE "Purity/0FRt0SRqqt0FBt/event_purity"trainer/global_step2:JPÈ aÄNé bI &Purity/0FRt2SRqqt2FBt/event_proportion"trainer/global_step2:JP7€ÎºJê bE "Purity/1FRt1SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPž\ÒÈHë bC Purity/2FRt*SRqqt*FBt/FBt_purity"trainer/global_step2:JP¥÷}cNì bI &Purity/4FRt1SRqqt1FBt/event_proportion"trainer/global_step2:JP=Ϲ£Jí bE "Purity/3FRt1SRqqt1FBt/event_purity"trainer/global_step2:JPH{ÏéJî bE "Purity/*FRt3SRqqt*FBt/event_purity"trainer/global_step2:JP‰:€%Hï bC Purity/2FRt*SRqqt0FBt/FRt_purity"trainer/global_step2:JP¬É¿šNð bI &Purity/2FRt0SRqqt4FBt/event_proportion"trainer/global_step2:JPÔtƒ,Jñ bE "Purity/2FRt1SRqqt*FBt/event_purity"trainer/global_step2:JPø©øHNò bI &Purity/3FRt0SRqqt2FBt/event_proportion"trainer/global_step2:JP7ü0Nó bI &Purity/4FRt3SRqqt0FBt/event_proportion"trainer/global_step2:JP |éJô bE "Purity/0FRt4SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPD°€¹@õ b; particle/accuracy_6_of_6"trainer/global_step2:JP;Q}Jö bE "Purity/*FRt1SRqqt*FBt/event_purity"trainer/global_step2:JPP“ΰN÷ bI &Purity/1FRt2SRqqt0FBt/event_proportion"trainer/global_step2:JP;r'Nø bI &Purity/2FRt0SRqqt3FBt/event_proportion"trainer/global_step2:JP¾o9ïNù bI &Purity/3FRt3SRqqt0FBt/event_proportion"trainer/global_step2:JPA/rNú bI &Purity/4FRt4SRqqt4FBt/event_proportion"trainer/global_step2:JP ͤJû bE "Purity/3FRt2SRqqt1FBt/event_purity"trainer/global_step2:JP0·¼CJü bE "Purity/0FRt4SRqqt0FBt/event_purity"trainer/global_step2:JPë=à8Hý bC Purity/*FRt1SRqqt4FBt/FBt_purity"trainer/global_step2:JP6œfˆJþ bE "Purity/*FRt2SRqqt*FBt/event_purity"trainer/global_step2:JPi~BMNÿ bI &Purity/0FRt3SRqqt1FBt/event_proportion"trainer/global_step2:JPeοŠN€ bI &Purity/2FRt*SRqqt4FBt/event_proportion"trainer/global_step2:JP‰œ¯H bC Purity/2FRt0SRqqt2FBt/FBt_purity"trainer/global_step2:JPPJÔ6J‚ bE "Purity/2FRt2SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPE÷ÿ@ƒ b; particle/accuracy_3_of_6"trainer/global_step2:JP¸@Ï­;„ b6 jet/accuracy_3_of_7"trainer/global_step2:JPç‰ø‹J… bE "Purity/1FRt*SRqqt4FBt/SRqqt_purity"trainer/global_step2:JP¡Î:J† bE "Purity/3FRt*SRqqt*FBt/event_purity"trainer/global_step2:JPSŽH‡ bC Purity/3FRt0SRqqt1FBt/FBt_purity"trainer/global_step2:JPqT ;ˆ b6 jet/accuracy_2_of_6"trainer/global_step2:JPÊ>‰P;‰ b6 jet/accuracy_5_of_7"trainer/global_step2:JPÁJŠ bE "Purity/0FRt*SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPî ?jH‹ bC Purity/1FRt1SRqqt2FBt/FRt_purity"trainer/global_step2:JPº‰QQJŒ bE "Purity/2FRt*SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP7©/ñN bI &Purity/2FRt0SRqqt1FBt/event_proportion"trainer/global_step2:JPݯW$NŽ bI &Purity/2FRt2SRqqt2FBt/event_proportion"trainer/global_step2:JP½0úñJ bE "Purity/*FRt1SRqqt3FBt/SRqqt_purity"trainer/global_step2:JP ÀÖ®J bE "Purity/3FRt3SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPQÑU|J‘ bE "Purity/1FRt1SRqqt3FBt/SRqqt_purity"trainer/global_step2:JPß,¢1H’ bC Purity/1FRt*SRqqt3FBt/FRt_purity"trainer/global_step2:JPå‘ìH“ bC Purity/1FRt0SRqqt3FBt/FBt_purity"trainer/global_step2:JPŽQ²ãJ” bE "Purity/*FRt1SRqqt3FBt/event_purity"trainer/global_step2:JPstôH• bC Purity/0FRt1SRqqt3FBt/FBt_purity"trainer/global_step2:JPÁT¨J– bE "Purity/1FRt1SRqqt3FBt/event_purity"trainer/global_step2:JP-f§H— bC Purity/*FRt0SRqqt2FBt/FRt_purity"trainer/global_step2:JPOûN˜ bI &Purity/*FRt4SRqqt4FBt/event_proportion"trainer/global_step2:JPד5J™ bE "Purity/0FRt1SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPǸyJš bE "Purity/1FRt3SRqqt*FBt/event_purity"trainer/global_step2:JPüÈÀ£J› bE "Purity/3FRt2SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPVó#[Nœ bI &Purity/3FRt2SRqqt2FBt/event_proportion"trainer/global_step2:JP$æÚH bC Purity/3FRt0SRqqt*FBt/FRt_purity"trainer/global_step2:JP¨HGJž bE "Purity/1FRt4SRqqt0FBt/event_purity"trainer/global_step2:JP:0K!NŸ bI &Purity/4FRt*SRqqt0FBt/event_proportion"trainer/global_step2:JPЦÓiN  bI &Purity/4FRt2SRqqt0FBt/event_proportion"trainer/global_step2:JPþÏDJ¡ bE "Purity/0FRt2SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPÓ 2(J¢ bE "Purity/*FRt3SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP“³ÅN£ bI &Purity/*FRt0SRqqt*FBt/event_proportion"trainer/global_step2:JP‡uiYH¤ bC Purity/1FRt2SRqqt0FBt/FRt_purity"trainer/global_step2:JP=}EH¥ bC Purity/*FRt2SRqqt3FBt/FBt_purity"trainer/global_step2:JPÓ#q¶J¦ bE "Purity/0FRt1SRqqt*FBt/event_purity"trainer/global_step2:JP•¥ÎÜN§ bI &Purity/0FRt3SRqqt4FBt/event_proportion"trainer/global_step2:JPÜ?ã·J¨ bE "Purity/1FRt3SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPçå›mJ© bE "Purity/3FRt2SRqqt*FBt/event_purity"trainer/global_step2:JP Aç[Nª bI &Purity/3FRt3SRqqt3FBt/event_proportion"trainer/global_step2:JP­³—N« bI &Purity/4FRt3SRqqt*FBt/event_proportion"trainer/global_step2:JP£D+J¬ bE "Purity/*FRt*SRqqt0FBt/event_purity"trainer/global_step2:JPTÆ0!H­ bC Purity/*FRt0SRqqt3FBt/FBt_purity"trainer/global_step2:JPʈë N® bI &Purity/0FRt0SRqqt4FBt/event_proportion"trainer/global_step2:JPв¤´J¯ bE "Purity/0FRt2SRqqt*FBt/event_purity"trainer/global_step2:JPÆ.c«H° bC Purity/2FRt2SRqqt1FBt/FRt_purity"trainer/global_step2:JPû÷TÅN± bI &Purity/3FRt3SRqqt*FBt/event_proportion"trainer/global_step2:JPÞ:NÎH² bC Purity/*FRt2SRqqt2FBt/FRt_purity"trainer/global_step2:JPa”ØìJ³ bE "Purity/0FRt3SRqqt1FBt/event_purity"trainer/global_step2:JPÙe‘nH´ bC Purity/*FRt1SRqqt2FBt/FBt_purity"trainer/global_step2:JP±ñ5Hµ bC Purity/1FRt0SRqqt2FBt/FRt_purity"trainer/global_step2:JPÕ]¬LJ¶ bE "Purity/2FRt*SRqqt*FBt/event_purity"trainer/global_step2:JPîm'mN· bI &Purity/4FRt1SRqqt0FBt/event_proportion"trainer/global_step2:JP,Ü3ŠJ¸ bE "Purity/1FRt1SRqqt4FBt/SRqqt_purity"trainer/global_step2:JP™2 J¹ bE "Purity/3FRt3SRqqt*FBt/event_purity"trainer/global_step2:JPX¦ëHº bC Purity/*FRt*SRqqt1FBt/FBt_purity"trainer/global_step2:JPهћN» bI &Purity/*FRt4SRqqt1FBt/event_proportion"trainer/global_step2:JPø€ùJ¼ bE "Purity/0FRt1SRqqt2FBt/event_purity"trainer/global_step2:JPÖ:y+H½ bC Purity/1FRt3SRqqt0FBt/FRt_purity"trainer/global_step2:JP;QóCN¾ bI &Purity/3FRt3SRqqt2FBt/event_proportion"trainer/global_step2:JPûXhžN¿ bI &Purity/3FRt4SRqqt4FBt/event_proportion"trainer/global_step2:JPŠNÅHÀ bC Purity/1FRt1SRqqt3FBt/FBt_purity"trainer/global_step2:JPšè"JÁ bE "Purity/*FRt2SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP}›ìJ bE "Purity/0FRt*SRqqt2FBt/event_purity"trainer/global_step2:JP_ȃ/Nà bI &Purity/0FRt3SRqqt0FBt/event_proportion"trainer/global_step2:JP ‡»ÖNÄ bI &Purity/2FRt3SRqqt*FBt/event_proportion"trainer/global_step2:JPYµÄìHÅ bC Purity/3FRt1SRqqt*FBt/FBt_purity"trainer/global_step2:JPD=ÙJÆ bE "Purity/0FRt0SRqqt4FBt/event_purity"trainer/global_step2:JPô(HÇ bC Purity/3FRt2SRqqt*FBt/FBt_purity"trainer/global_step2:JP€UéÈHÈ bC Purity/2FRt3SRqqt1FBt/FRt_purity"trainer/global_step2:JPpô=JÉ bE "Purity/0FRt2SRqqt0FBt/event_purity"trainer/global_step2:JPÂpnHÊ bC Purity/1FRt*SRqqt1FBt/FRt_purity"trainer/global_step2:JP9MöJË bE "Purity/3FRt*SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP?îþ8Ì b3 particle/f_score"trainer/global_step2:JPN›ÎŽHÍ bC Purity/*FRt3SRqqt*FBt/FBt_purity"trainer/global_step2:JPDí©Ö@Î b; particle/accuracy_7_of_7"trainer/global_step2:JP­ ÙHÏ bC Purity/4FRt0SRqqt*FBt/FRt_purity"trainer/global_step2:JPƒš&HÐ bC Purity/*FRt*SRqqt*FBt/FRt_purity"trainer/global_step2:JP&õENÑ bI &Purity/*FRt*SRqqt1FBt/event_proportion"trainer/global_step2:JP=Ÿd€JÒ bE "Purity/*FRt3SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPÅþzJÓ bE "Purity/0FRt1SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPrÁ~JÔ bE "Purity/2FRt1SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP#Ëö3HÕ bC Purity/2FRt1SRqqt2FBt/FBt_purity"trainer/global_step2:JPF¶wrJÖ bE "Purity/0FRt4SRqqt*FBt/event_purity"trainer/global_step2:JPnú+@× b; particle/accuracy_6_of_7"trainer/global_step2:JPºÿ\JØ bE "Purity/*FRt1SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP¿…ANÙ bI &Purity/*FRt2SRqqt*FBt/event_proportion"trainer/global_step2:JPk,H{NÚ bI &Purity/0FRt0SRqqt0FBt/event_proportion"trainer/global_step2:JP ì¹NÛ bI &Purity/0FRt0SRqqt2FBt/event_proportion"trainer/global_step2:JP.VNÜ bI &Purity/0FRt0SRqqt3FBt/event_proportion"trainer/global_step2:JP7Ÿb[JÝ bE "Purity/1FRt*SRqqt3FBt/event_purity"trainer/global_step2:JPÙó°æJÞ bE "Purity/1FRt*SRqqt3FBt/SRqqt_purity"trainer/global_step2:JP@# #Hß bC Purity/4FRt*SRqqt0FBt/FRt_purity"trainer/global_step2:JP î¼Jà bE "Purity/*FRt0SRqqt2FBt/event_purity"trainer/global_step2:JPï[‚Ná bI &Purity/*FRt0SRqqt3FBt/event_proportion"trainer/global_step2:JPGýhÔHâ bC Purity/*FRt1SRqqt*FBt/FRt_purity"trainer/global_step2:JPX¨öNã bI &Purity/3FRt4SRqqt0FBt/event_proportion"trainer/global_step2:JPsïJä bE "Purity/1FRt2SRqqt2FBt/SRqqt_purity"trainer/global_step2:JP[f¦pJå bE "Purity/3FRt1SRqqt2FBt/SRqqt_purity"trainer/global_step2:JPþ>ï_Jæ bE "Purity/2FRt2SRqqt2FBt/SRqqt_purity"trainer/global_step2:JP8g™¥Jç bE "Purity/0FRt*SRqqt0FBt/event_purity"trainer/global_step2:JPè jJè bE "Purity/0FRt1SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP$p.Jé bE "Purity/1FRt1SRqqt*FBt/event_purity"trainer/global_step2:JPÅ gWNê bI &Purity/2FRt1SRqqt1FBt/event_proportion"trainer/global_step2:JPÿËcJë bE "Purity/2FRt1SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPõ¾@Hì bC Purity/2FRt1SRqqt1FBt/FBt_purity"trainer/global_step2:JP䥨¹Ní bI &Purity/3FRt3SRqqt4FBt/event_proportion"trainer/global_step2:JPç½9=Nî bI &Purity/3FRt4SRqqt3FBt/event_proportion"trainer/global_step2:JP2¼AJï bE "Purity/*FRt0SRqqt0FBt/event_purity"trainer/global_step2:JPë½·VHð bC Purity/*FRt3SRqqt0FBt/FRt_purity"trainer/global_step2:JP:¤Ò·Nñ bI &Purity/*FRt4SRqqt3FBt/event_proportion"trainer/global_step2:JP@´P@Nò bI &Purity/1FRt3SRqqt2FBt/event_proportion"trainer/global_step2:JP1õÖ×Nó bI &Purity/3FRt0SRqqt*FBt/event_proportion"trainer/global_step2:JP6Ù90Nˆ bI &Purity/4FRt*SRqqt3FBt/event_proportion"trainer/global_step2:JP›sEêH‰ bC Purity/2FRt0SRqqt3FBt/FRt_purity"trainer/global_step2:JP†Ëí`JŠ bE "Purity/3FRt3SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPÖ±m»J‹ bE "Purity/*FRt2SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP„?«eJŒ bE "Purity/0FRt0SRqqt1FBt/event_purity"trainer/global_step2:JPÍÍäJ bE "Purity/1FRt*SRqqt2FBt/SRqqt_purity"trainer/global_step2:JP¶Ùá¯HŽ bC Purity/3FRt3SRqqt*FBt/FRt_purity"trainer/global_step2:JPÁyàxH bC Purity/*FRt*SRqqt1FBt/FRt_purity"trainer/global_step2:JP 3?N bI &Purity/*FRt0SRqqt1FBt/event_proportion"trainer/global_step2:JPµ"!N‘ bI &Purity/1FRt0SRqqt4FBt/event_proportion"trainer/global_step2:JP2 H’ bC Purity/*FRt0SRqqt4FBt/FRt_purity"trainer/global_step2:JPƒ‘y¶J“ bE "Purity/*FRt1SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPéÙðN” bI &Purity/0FRt3SRqqt3FBt/event_proportion"trainer/global_step2:JPÞúg@• b; particle/accuracy_3_of_4"trainer/global_step2:JP/,f H– bC Purity/0FRt*SRqqt4FBt/FBt_purity"trainer/global_step2:JPƒwÇ‘N— bI &Purity/*FRt3SRqqt2FBt/event_proportion"trainer/global_step2:JP˜ÛÊN˜ bI &Purity/0FRt*SRqqt1FBt/event_proportion"trainer/global_step2:JPËh—îN™ bI &Purity/1FRt1SRqqt*FBt/event_proportion"trainer/global_step2:JPˆ¯¸Hš bC Purity/2FRt*SRqqt2FBt/FRt_purity"trainer/global_step2:JPºû® H› bC Purity/2FRt2SRqqt1FBt/FBt_purity"trainer/global_step2:JP¾½ÓDHœ bC Purity/2FRt3SRqqt*FBt/FRt_purity"trainer/global_step2:JPõ§µþN bI &Purity/4FRt*SRqqt*FBt/event_proportion"trainer/global_step2:JPólKµ;ž b6 jet/accuracy_3_of_3"trainer/global_step2:JP·s8HŸ bC Purity/1FRt*SRqqt*FBt/FRt_purity"trainer/global_step2:JP±×ÐtJ  bE "Purity/1FRt1SRqqt0FBt/event_purity"trainer/global_step2:JPZZLWN¡ bI &Purity/2FRt*SRqqt2FBt/event_proportion"trainer/global_step2:JP²8ÞJ¢ bE "Purity/2FRt0SRqqt*FBt/event_purity"trainer/global_step2:JPšGFÈJ£ bE "Purity/0FRt*SRqqt4FBt/event_purity"trainer/global_step2:JP«e¡H¤ bC Purity/1FRt1SRqqt4FBt/FRt_purity"trainer/global_step2:JP`åŠH¥ bC Purity/*FRt0SRqqt0FBt/FRt_purity"trainer/global_step2:JPÄÛïN¦ bI &Purity/1FRt3SRqqt0FBt/event_proportion"trainer/global_step2:JP›uþJ§ bE "Purity/2FRt3SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPp£ìN¨ bI &Purity/4FRt0SRqqt2FBt/event_proportion"trainer/global_step2:JP6ˆ @© b; particle/accuracy_4_of_5"trainer/global_step2:JPÀ‹ëùHª bC Purity/0FRt0SRqqt*FBt/FBt_purity"trainer/global_step2:JP»°v‹N« bI &Purity/0FRt2SRqqt0FBt/event_proportion"trainer/global_step2:JP£ù-N¬ bI &Purity/1FRt2SRqqt1FBt/event_proportion"trainer/global_step2:JPóãÓ N­ bI &Purity/2FRt1SRqqt2FBt/event_proportion"trainer/global_step2:JPµÚñá;® b6 jet/accuracy_4_of_4"trainer/global_step2:JPŠx²¦@¯ b; particle/accuracy_2_of_2"trainer/global_step2:JPØÐ l@° b; particle/accuracy_1_of_6"trainer/global_step2:JP9$½H± bC Purity/0FRt2SRqqt2FBt/FBt_purity"trainer/global_step2:JPg­LH² bC Purity/1FRt*SRqqt1FBt/FBt_purity"trainer/global_step2:JP7ÚkòH³ bC Purity/1FRt1SRqqt0FBt/FRt_purity"trainer/global_step2:JPsòÅqN´ bI &Purity/2FRt2SRqqt4FBt/event_proportion"trainer/global_step2:JPæ5á:Hµ bC Purity/1FRt2SRqqt2FBt/FRt_purity"trainer/global_step2:JPiðN¶ bI &Purity/0FRt*SRqqt3FBt/event_proportion"trainer/global_step2:JP!?¨ÑH· bC Purity/2FRt2SRqqt*FBt/FRt_purity"trainer/global_step2:JPJõ#³J¸ bE "Purity/3FRt2SRqqt0FBt/event_purity"trainer/global_step2:JP<V.J¹ bE "Purity/3FRt*SRqqt1FBt/event_purity"trainer/global_step2:JPkc²º bE "Purity/2FRt4SŒµ)ã6Rqqt0FBt/SRqqt_purity"trainer/global_step2:JPYô•gH» bC Purity/2FRt3SRqqt1FBt/FBt_purity"trainer/global_step2:JPò¼J¼ bE "Purity/3FRt4SRqqt*FBt/event_purity"trainer/global_step2:JPïñ$#H½ bC Purity/1FRt4SRqqt0FBt/FRt_purity"trainer/global_step2:JPè`w;¾ b6 loss/val_total_loss"trainer/global_step2:JP¢s‚ŠJ¿ bE "Purity/*FRt1SRqqt1FBt/event_purity"trainer/global_step2:JPß7ÝjNÀ bI &Purity/0FRt1SRqqt1FBt/event_proportion"trainer/global_step2:JPrÑ}üHÁ bC Purity/0FRt1SRqqt1FBt/FBt_purity"trainer/global_step2:JPÿ¸=iN bI &Purity/0FRt3SRqqt*FBt/event_proportion"trainer/global_step2:JP))INà bI &Purity/1FRt*SRqqt2FBt/event_proportion"trainer/global_step2:JPÞ#³HÄ bC Purity/1FRt0SRqqt1FBt/FBt_purity"trainer/global_step2:JP†-ífJÅ bE "Purity/1FRt2SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPâ6MHÆ bC Purity/*FRt*SRqqt2FBt/FRt_purity"trainer/global_step2:JPûË:FNÇ bI &Purity/3FRt3SRqqt1FBt/event_proportion"trainer/global_step2:JPî$ÈÏNÈ bI &Purity/4FRt3SRqqt3FBt/event_proportion"trainer/global_step2:JPõNÉ bI &Purity/2FRt0SRqqt0FBt/event_proportion"trainer/global_step2:JP# õHÊ bC Purity/*FRt2SRqqt1FBt/FBt_purity"trainer/global_step2:JPk:MüHË bC Purity/0FRt*SRqqt*FBt/FBt_purity"trainer/global_step2:JPÿÊ HÌ bC Purity/2FRt*SRqqt1FBt/FRt_purity"trainer/global_step2:JP$¦³ÚJÍ bE "Purity/3FRt1SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP] HÎ bC Purity/4FRt2SRqqt*FBt/FRt_purity"trainer/global_step2:JPãiqHÏ bC Purity/1FRt2SRqqt1FBt/FRt_purity"trainer/global_step2:JP¼·ÊžJÐ bE "Purity/1FRt3SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPì¼ÎNÑ bI &Purity/3FRt2SRqqt3FBt/event_proportion"trainer/global_step2:JPÍ?5ÂHÒ bC Purity/1FRt0SRqqt*FBt/FRt_purity"trainer/global_step2:JPò?‰NÓ bI &Purity/4FRt2SRqqt4FBt/event_proportion"trainer/global_step2:JPJ*ÍãHÔ bC Purity/*FRt2SRqqt*FBt/FRt_purity"trainer/global_step2:JP/IåNÕ bI &Purity/*FRt3SRqqt3FBt/event_proportion"trainer/global_step2:JPá êNÖ bI &Purity/1FRt3SRqqt1FBt/event_proportion"trainer/global_step2:JPűÀîJ× bE "Purity/2FRt*SRqqt2FBt/event_purity"trainer/global_step2:JPö‚©9NØ bI &Purity/2FRt2SRqqt3FBt/event_proportion"trainer/global_step2:JPcS•2NÙ bI &Purity/2FRt3SRqqt3FBt/event_proportion"trainer/global_step2:JPÊV×NÚ bI &Purity/3FRt*SRqqt0FBt/event_proportion"trainer/global_step2:JPÌÔ©NÛ bI &Purity/*FRt1SRqqt2FBt/event_proportion"trainer/global_step2:JP©‰³JÜ bE "Purity/*FRt1SRqqt2FBt/SRqqt_purity"trainer/global_step2:JPꢪINÝ bI &Purity/0FRt4SRqqt*FBt/event_proportion"trainer/global_step2:JPºxüÑ;Þ b6 jet/accuracy_1_of_4"trainer/global_step2:JPÑd`ÞJß bE "Purity/3FRt*SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPÕ_ ªHà bC Purity/0FRt0SRqqt4FBt/FBt_purity"trainer/global_step2:JPü~á6Há bC Purity/3FRt*SRqqt2FBt/FBt_purity"trainer/global_step2:JP/·òJâ bE "Purity/1FRt0SRqqt1FBt/event_purity"trainer/global_step2:JP¾½’CJã bE "Purity/0FRt4SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPaÔŸ Jä bE "Purity/2FRt3SRqqt1FBt/event_purity"trainer/global_step2:JPlš’Hå bC Purity/3FRt4SRqqt0FBt/FRt_purity"trainer/global_step2:JPT •>Hæ bC Purity/2FRt0SRqqt3FBt/FBt_purity"trainer/global_step2:JPt€IkJç bE "Purity/*FRt*SRqqt3FBt/SRqqt_purity"trainer/global_step2:JPTiîlJè bE "Purity/*FRt*SRqqt3FBt/event_purity"trainer/global_step2:JPjºJé bE "Purity/0FRt*SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP]VŽðNê bI &Purity/1FRt*SRqqt0FBt/event_proportion"trainer/global_step2:JP"ÜãNë bI &Purity/3FRt1SRqqt4FBt/event_proportion"trainer/global_step2:JPW tHì bC Purity/3FRt*SRqqt1FBt/FBt_purity"trainer/global_step2:JPÑ›=Hí bC Purity/*FRt0SRqqt4FBt/FBt_purity"trainer/global_step2:JP_ã—¾@î b; particle/accuracy_1_of_7"trainer/global_step2:JPz”rHï bC Purity/0FRt3SRqqt*FBt/FBt_purity"trainer/global_step2:JPTVÔJð bE "Purity/2FRt2SRqqt0FBt/event_purity"trainer/global_step2:JPÀÊëJñ bE "Purity/2FRt3SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPÔ=XšNò bI &Purity/2FRt3SRqqt0FBt/event_proportion"trainer/global_step2:JPš‹¨^Jó bE "Purity/*FRt3SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPê¤âYJô bE "Purity/*FRt4SRqqt*FBt/event_purity"trainer/global_step2:JPDÅHUJõ bE "Purity/1FRt3SRqqt2FBt/event_purity"trainer/global_step2:JPŠ}§_Nö bI &Purity/*FRt1SRqqt4FBt/event_proportion"trainer/global_step2:JPÏ9žÐN÷ bI &Purity/*FRt3SRqqt4FBt/event_proportion"trainer/global_step2:JPìÒ< Hø bC Purity/2FRt1SRqqt2FBt/FRt_purity"trainer/global_step2:JPf…Ì5Nù bI &Purity/4FRt4SRqqt3FBt/event_proportion"trainer/global_step2:JP¸±·¼Hú bC Purity/0FRt4SRqqt*FBt/FBt_purity"trainer/global_step2:JP†¤ÜýHû bC Purity/1FRt1SRqqt4FBt/FBt_purity"trainer/global_step2:JPÙ_EŸNü bI &Purity/0FRt0SRqqt*FBt/event_proportion"trainer/global_step2:JP`ÓtJý bE "Purity/1FRt0SRqqt*FBt/event_purity"trainer/global_step2:JPÇùƒ‘Nþ bI &Purity/2FRt3SRqqt2FBt/event_proportion"trainer/global_step2:JPÍB^fHÿ bC Purity/3FRt*SRqqt*FBt/FRt_purity"trainer/global_step2:JP.1PoH€ bC Purity/*FRt2SRqqt2FBt/FBt_purity"trainer/global_step2:JPW`­ÖH bC Purity/4FRt1SRqqt*FBt/FRt_purity"trainer/global_step2:JP£Áã‹N‚ bI &Purity/*FRt1SRqqt1FBt/event_proportion"trainer/global_step2:JP.†¦®Nƒ bI &Purity/3FRt4SRqqt1FBt/event_proportion"trainer/global_step2:JPKD“²N„ bI &Purity/4FRt*SRqqt4FBt/event_proportion"trainer/global_step2:JPð7%@… b; particle/accuracy_3_of_3"trainer/global_step2:JP¯:Ê™<† b7 particle/specificity"trainer/global_step2:JPóÞôG‡ bB validation_average_jet_accuracy"trainer/global_step2:JP ÁÃNˆ bI &Purity/1FRt0SRqqt*FBt/event_proportion"trainer/global_step2:JPy¬†J‰ bE "Purity/0FRt0SRqqt3FBt/event_purity"trainer/global_step2:JPø‚JŠ bE "Purity/*FRt*SRqqt4FBt/event_purity"trainer/global_step2:JPf¤'J‹ bE "Purity/1FRt4SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPÙÄÑ{JŒ bE "Purity/0FRt1SRqqt3FBt/SRqqt_purity"trainer/global_step2:JPUãÏH bC Purity/*FRt1SRqqt2FBt/FRt_purity"trainer/global_step2:JP‰j±bHŽ bC Purity/0FRt1SRqqt2FBt/FBt_purity"trainer/global_step2:JPDêóH bC Purity/1FRt0SRqqt2FBt/FBt_purity"trainer/global_step2:JP¦»J bE "Purity/*FRt3SRqqt0FBt/event_purity"trainer/global_step2:JP©Ô ~J‘ bE "Purity/3FRt0SRqqt1FBt/event_purity"trainer/global_step2:JP^…†÷J’ bE "Purity/0FRt1SRqqt3FBt/event_purity"trainer/global_step2:JPµg‡J“ bE "Purity/*FRt*SRqqt2FBt/SRqqt_purity"trainer/global_step2:JPš=3¼N” bI &Purity/1FRt1SRqqt1FBt/event_proportion"trainer/global_step2:JPЫ!úN• bI &Purity/3FRt4SRqqt*FBt/event_proportion"trainer/global_step2:JP9AH– bC Purity/3FRt1SRqqt1FBt/FRt_purity"trainer/global_step2:JP :G™H— bC Purity/3FRt2SRqqt1FBt/FRt_purity"trainer/global_step2:JP™›†ÐH˜ bC Purity/2FRt3SRqqt*FBt/FBt_purity"trainer/global_step2:JPÏÁº#H™ bC Purity/*FRt1SRqqt3FBt/FBt_purity"trainer/global_step2:JP·f´ÉJš bE "Purity/*FRt2SRqqt3FBt/SRqqt_purity"trainer/global_step2:JPb³Ò‡J› bE "Purity/*FRt*SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPhƒÇHœ bC Purity/1FRt*SRqqt*FBt/FBt_purity"trainer/global_step2:JPiкSH bC Purity/1FRt1SRqqt2FBt/FBt_purity"trainer/global_step2:JPAöµNž bI &Purity/1FRt4SRqqt*FBt/event_proportion"trainer/global_step2:JP¥Ùt”NŸ bI &Purity/2FRt4SRqqt*FBt/event_proportion"trainer/global_step2:JPÇb}H  bC Purity/*FRt3SRqqt1FBt/FRt_purity"trainer/global_step2:JPÖî…#J¡ bE "Purity/*FRt2SRqqt3FBt/event_purity"trainer/global_step2:JP)Ï&ÚH¢ bC Purity/0FRt0SRqqt1FBt/FBt_purity"trainer/global_step2:JP}ê‚N£ bI &Purity/2FRt*SRqqt0FBt/event_proportion"trainer/global_step2:JPig©®J¤ bE "Purity/2FRt0SRqqt1FBt/event_purity"trainer/global_step2:JPù¿6×N¥ bI &Purity/4FRt4SRqqt0FBt/event_proportion"trainer/global_step2:JPªÿ($;¦ b6 jet/accuracy_1_of_6"trainer/global_step2:JP'Ä!\;§ b6 jet/accuracy_6_of_6"trainer/global_step2:JP Ü>ñ;¨ b6 jet/accuracy_4_of_7"trainer/global_step2:JP ‹0J© bE "Purity/2FRt0SRqqt3FBt/event_purity"trainer/global_step2:JPƒ—ü"Hª bC Purity/*FRt0SRqqt2FBt/FBt_purity"trainer/global_step2:JPl³GJN« bI &Purity/0FRt0SRqqt1FBt/event_proportion"trainer/global_step2:JP ùÀ«N¬ bI &Purity/0FRt1SRqqt3FBt/event_proportion"trainer/global_step2:JPªZC%H­ bC Purity/3FRt3SRqqt0FBt/FRt_purity"trainer/global_step2:JPñ¿›uN® bI &Purity/*FRt*SRqqt3FBt/event_proportion"trainer/global_step2:JP ãóN¯ bI &Purity/2FRt1SRqqt4FBt/event_proportion"trainer/global_step2:JPŸYJ° bE "Purity/2FRt3SRqqt*FBt/event_purity"trainer/global_step2:JP’ƒ8eKH¶ bC Purity/1FRt0SRqqt4FBt/FBt_purity"trainer/global_step2:JPÆïAJ· bE "Purity/*FRt*SRqqt1FBt/SRqqt_purity"trainer/global_step2:JP`àçN¸ bI &Purity/0FRt4SRqqt0FBt/event_proportion"trainer/global_step2:JPÓ©þ¾N¹ bI &Purity/3FRt2SRqqt1FBt/event_proportion"trainer/global_step2:JP*uÝDJº bE "Purity/0FRt*SRqqt*FBt/event_purity"trainer/global_step2:JP„ ›UN» bI &Purity/0FRt*SRqqt2FBt/event_proportion"trainer/global_step2:JP»ÊÀN¼ bI &Purity/0FRt1SRqqt0FBt/event_proportion"trainer/global_step2:JPQ¦þØH½ bC Purity/2FRt0SRqqt2FBt/FRt_purity"trainer/global_step2:JP€£fõ@¾ b; particle/accuracy_1_of_1"trainer/global_step2:JPäÜ”úJ¿ bE "Purity/0FRt4SRqqt1FBt/event_purity"trainer/global_step2:JPÐ>à³NÀ bI &Purity/0FRt4SRqqt1FBt/event_proportion"trainer/global_step2:JPŠ'¥NÁ bI &Purity/3FRt0SRqqt1FBt/event_proportion"trainer/global_step2:JPÖi.@ b; particle/accuracy_2_of_3"trainer/global_step2:JPÑQ^ë@à b; particle/accuracy_3_of_5"trainer/global_step2:JPBuûHÄ bC Purity/1FRt3SRqqt1FBt/FRt_purity"trainer/global_step2:JP¯O.­HÅ bC Purity/*FRt0SRqqt*FBt/FRt_purity"trainer/global_step2:JP·ê (JÆ bE "Purity/0FRt*SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPô©iJÇ bE "Purity/0FRt1SRqqt0FBt/event_purity"trainer/global_step2:JPf]¿„NÈ bI &Purity/1FRt4SRqqt0FBt/event_proportion"trainer/global_step2:JPZÔè NÉ bI &Purity/2FRt4SRqqt4FBt/event_proportion"trainer/global_step2:JP@K·+NÊ bI &Purity/4FRt*SRqqt1FBt/event_proportion"trainer/global_step2:JPÚh@Ë b; particle/accuracy_4_of_4"trainer/global_step2:JPPö9JÌ bE "Purity/*FRt2SRqqt2FBt/event_purity"trainer/global_step2:JPlæ¥NÍ bI &Purity/1FRt1SRqqt4FBt/event_proportion"trainer/global_step2:JP‰¯2%JÎ bE "Purity/1FRt2SRqqt1FBt/event_purity"trainer/global_step2:JPá­€ÊHÏ bC Purity/2FRt2SRqqt0FBt/FRt_purity"trainer/global_step2:JPÙùErHÐ bC Purity/*FRt4SRqqt*FBt/FBt_purity"trainer/global_step2:JPH,òìHÑ bC Purity/1FRt*SRqqt4FBt/FRt_purity"trainer/global_step2:JP¶7!JÒ bE "Purity/1FRt0SRqqt4FBt/event_purity"trainer/global_step2:JP–üGHÓ bC Purity/1FRt4SRqqt*FBt/FRt_purity"trainer/global_step2:JPÐuØäHÔ bC Purity/3FRt0SRqqt2FBt/FRt_purity"trainer/global_step2:JPwÄ NÕ bI &Purity/*FRt0SRqqt2FBt/event_proportion"trainer/global_step2:JPžžJÖ bE "Purity/*FRt0SRqqt3FBt/event_purity"trainer/global_step2:JPŸÌ*iJ× bE "Purity/1FRt*SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP„ý’HØ bC Purity/1FRt1SRqqt*FBt/FRt_purity"trainer/global_step2:JP|_°×2Ù - _wandbruntime‚1427  _runtime‚1427«®MùÁMÚ Á¦ @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.18421052396297455 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.3571428656578064 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 +jet/accuracy_2_of_5‚0.37919867038726807 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 ,"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.625 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 epoch‚0 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9879372715950012 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9944315552711487 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 +jet/accuracy_3_of_5‚0.07282385975122452 +"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.25 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 & Purity/3FRt4SRqqt*FBt/FRt_purity‚0 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.3177083432674408 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 +jet/accuracy_2_of_2‚0.07419481873512268 ("Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0 ( Purity/*FRt1SRqqt3FBt/FRt_purity‚0.5 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9845969676971436 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9868807792663574 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 :"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.43886834383010864 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.3741109371185303 particle/accuracy_2_of_5‚1 jet/accuracy_7_of_7‚0 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_2_of_4‚1 *"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.5 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9837027192115784 :"Purity/*FRt1SRqqt2FBt/event_purity‚0.03353174775838852 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.4264393150806427 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.1277281492948532 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9259259700775146 ( Purity/1FRt0SRqqt3FBt/FRt_purity‚0.5 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.6666666865348816 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 :"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.37385621666908264 * Purity/4FRt*SRqqt*FBt/FRt_purity‚0.125 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 ("Purity/0FRt2SRqqt3FBt/SRqqt_purity‚1 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6236502528190613 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.3214111626148224 (particle/accuracy‚0.8375029563903809 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 ("Purity/4FRt*SRqqt*FBt/SRqqt_purity‚1 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚1 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 -"Purity/1FRt0SRqqt3FBt/event_purity‚0.0625 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.18850575387477875 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9937499761581421 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 8 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.35170507431030273 ;"Purity/*FRt2SRqqt0FBt/event_purity‚0.018949981778860092 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 ( Purity/*FRt*SRqqt4FBt/FRt_purity‚0.5 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.3769841492176056 *jet/accuracy_1_of_3‚0.8083683252334595 +jet/accuracy_3_of_6‚0.03846153989434242 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.13219207525253296 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 & Purity/*FRt1SRqqt4FBt/FRt_purity‚1 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.0623522587120533 7 Purity/*FRt3SRqqt1FBt/FBt_purity‚0.9629629850387573 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.4416269063949585 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 jet/accuracy_5_of_6‚0 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9856921434402466 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6466619372367859 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.1380334496498108 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 _step‚98 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.13026821613311768 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 6 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.208333358168602 *"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.5 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.18227867782115936 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 :"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.43209877610206604 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 jet/accuracy_1_of_7‚1 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9976190328598022 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.1850256621837616 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.21549750864505768 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.20000000298023224 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 8"Purity/1FRt*SRqqt2FBt/event_purity‚0.054151501506567 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.11735910922288895 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.3020833432674408 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 particle/accuracy_1_of_3‚1 particle/accuracy_1_of_5‚1 7 Purity/*FRt*SRqqt4FBt/FBt_purity‚0.8333333134651184 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.4639628529548645 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9907218813896179 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 ("Purity/*FRt3SRqqt*FBt/event_purity‚0 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.1269349753856659 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 +"Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.25 particle/accuracy_6_of_6‚1 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.1870577037334442 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.015942851081490517 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.4298752546310425 particle/accuracy_3_of_6‚1 jet/accuracy_3_of_7‚0 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 +jet/accuracy_2_of_6‚0.32692307233810425 jet/accuracy_5_of_7‚0 :"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.40691903233528137 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.34062495827674866 :"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.41714853048324585 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.7222222089767456 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.4166666567325592 +"Purity/1FRt1SRqqt3FBt/SRqqt_purity‚0.75 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.4444444477558136 7 Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9166666865348816 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 7 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.9047619104385376 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 8 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.47931405901908875 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 :"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.43736281991004944 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 :"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.43965521454811096 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 7 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.1796296387910843 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 :"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.34185802936553955 :"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.31536099314689636 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.12314967811107635 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.35227179527282715 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 9"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.2995210587978363 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.1528686136007309 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9166667461395264 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 ;"Purity/0FRt2SRqqt*FBt/event_purity‚0.036365997046232224 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2916666567325592 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 * Purity/*FRt2SRqqt2FBt/FRt_purity‚0.375 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9687660336494446 6 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.523217499256134 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.004904275760054588 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9950029850006104 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 ;"Purity/0FRt1SRqqt2FBt/event_purity‚0.056342363357543945 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.18391813337802887 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.9166666865348816 :"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.36364561319351196 :"Purity/0FRt*SRqqt2FBt/event_purity‚0.22959132492542267 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.0833333358168602 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.04370106756687164 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.33812612295150757 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.4314594268798828 'particle/f_score‚0.3370826840400696 7 Purity/*FRt3SRqqt*FBt/FBt_purity‚0.9642857313156128 particle/accuracy_7_of_7‚1 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.19171233475208282 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 8"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.315907746553421 :"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.42424097657203674 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.4483811557292938 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9659090638160706 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 particle/accuracy_6_of_7‚1 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.4490540027618408 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.0555555559694767 +"Purity/1FRt*SRqqt3FBt/SRqqt_purity‚0.75 * Purity/4FRt*SRqqt0FBt/FRt_purity‚0.125 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.2150954306125641 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.18999351561069489 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.3166666626930237 +"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.75 *"Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0.5 :"Purity/0FRt*SRqqt0FBt/event_purity‚0.34765133261680603 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.4377758502960205 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.012812395580112934 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 8"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.474702388048172 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.09917915612459183 7 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.1842963546514511 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 *jet/accuracy_2_of_4‚0.2926087975502014 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.05519581213593483 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5169184803962708 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.10288066416978836 jet/accuracy_5_of_5‚0 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.13111110031604767 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9766128063201904 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.37950366735458374 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.5326389670372009 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.1904793530702591 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9924511313438416 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 <"Purity/2FRt*SRqqt0FBt/event_purity‚0.0015432098880410194 *"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.5 jet/accuracy_6_of_7‚0 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.33947011828422546 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.2666666805744171 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.4080445170402527 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.4629371464252472 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.3577527105808258 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 *jet/accuracy_2_of_3‚0.1956164389848709 - Purity/0FRt0SRqqt3FBt/FBt_purity‚0.921875 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 & Purity/1FRt2SRqqt2FBt/FBt_purity‚1 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 *jet/accuracy_1_of_2‚0.6960355043411255 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.9078947305679321 ;"Purity/1FRt1SRqqt1FBt/event_purity‚0.015317751094698906 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 8 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.32592591643333435 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.4166666567325592 ("Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.4094215929508209 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.20000000298023224 ("Purity/0FRt3SRqqt0FBt/event_purity‚0 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.15946808457374573 <"Purity/*FRt2SRqqt1FBt/event_purity‚0.0022675737272948027 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.10862766206264496 particle/accuracy_3_of_7‚1 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.08206985145807266 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 particle/accuracy_5_of_6‚1 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9864341616630554 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.00515872985124588 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 *jet/accuracy_1_of_1‚0.4453316330909729 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.29794153571128845 7 Purity/2FRt*SRqqt1FBt/FBt_purity‚0.9960317611694336 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 :"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.29629626870155334 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 +"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.75 7 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.1533239781856537 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022*üQ‡E particle/accuracy_5_of_5‚1 6 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.357394278049469 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.06818182021379471 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 :"Purity/0FRt0SRqqt2FBt/event_purity‚0.29543691873550415 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.4770068824291229 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 )jet/accuracy_3_of_4‚0.035651795566082 *jet/accuracy_1_of_5‚0.9234639406204224 9"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.3499999940395355 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.18291912972927094 :"Purity/2FRt0SRqqt2FBt/event_purity‚0.02631578966975212 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 ("Purity/0FRt3SRqqt*FBt/event_purity‚0 ( Purity/1FRt1SRqqt3FBt/FRt_purity‚0.5 _runtime‚1427.216771958 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚1 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 ( Purity/*FRt0SRqqt3FBt/FRt_purity‚0.5 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.12614542245864868 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.2305431067943573 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9965516924858093 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.0833333358168602 jet/accuracy_2_of_7‚0.5 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9737676382064819 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9809044003486633 particle/accuracy_1_of_4‚1 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.1666666716337204 particle/accuracy_2_of_7‚1 7 Purity/0FRt2SRqqt1FBt/FBt_purity‚0.9773584604263306 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.3747023940086365 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 trainer/global_step‚1208 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9868219494819641 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8032798767089844 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 9"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.3333333432674408 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 particle/accuracy_4_of_7‚1 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 particle/accuracy_2_of_6‚1 ("Purity/4FRt*SRqqt0FBt/SRqqt_purity‚1 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.3324505388736725 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.4166666567325592 :"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.36214855313301086 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9977704882621765 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.5222986340522766 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.0833333358168602 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.34009239077568054 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 & Purity/*FRt0SRqqt4FBt/FRt_purity‚0 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.4631842076778412 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 particle/accuracy_3_of_4‚1 + Purity/0FRt*SRqqt4FBt/FBt_purity‚0.8125 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 8 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.38081395626068115 6 Purity/2FRt2SRqqt1FBt/FBt_purity‚0.976190447807312 8 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.16981980204582214 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 +jet/accuracy_3_of_3‚0.01686221733689308 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.19672457873821259 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.012913917191326618 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.014109919779002666 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 & Purity/1FRt1SRqqt4FBt/FRt_purity‚1 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.11528763920068741 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 :"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.30454543232917786 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 particle/accuracy_4_of_5‚1 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9857938885688782 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 -jet/accuracy_4_of_4‚0.0009920635493472219 particle/accuracy_2_of_2‚1 particle/accuracy_1_of_6‚1 7 Purity/0FRt2SRqqt2FBt/FBt_purity‚0.9642857313156128 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9948357939720154 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.14059039950370789 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 8 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.36666667461395264 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.15158100426197052 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 +"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.25 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 & Purity/1FRt4SRqqt0FBt/FRt_purity‚0 *loss/val_total_loss‚50.640127528021544 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.0715567097067833 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 6 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.991266131401062 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 7 Purity/1FRt0SRqqt1FBt/FBt_purity‚0.9980308413505554 :"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.35905328392982483 8 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.43464696407318115 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9773243069648743 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9845403432846069 6 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.344946950674057 :"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.36666667461395264 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.27327588200569153 9"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.2927875220775604 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2154974788427353 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.15559065341949463 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 ;"Purity/2FRt*SRqqt2FBt/event_purity‚0.023255813866853714 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5327509045600891 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 *jet/accuracy_1_of_4‚0.8691949844360352 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.5263158082962036 + Purity/0FRt0SRqqt4FBt/FBt_purity‚0.8125 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.14590950310230255 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.2857142984867096 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 & Purity/3FRt4SRqqt0FBt/FRt_purity‚0 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 +"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.75 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.13475176692008972 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.4044511020183563 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8500000238418579 particle/accuracy_1_of_7‚1 7 Purity/0FRt3SRqqt*FBt/FBt_purity‚0.8888888955116272 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 8"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.286636620759964 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 :"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.26543211936950684 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 8 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.36742427945137024 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.12500807642936707 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 6 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.167772576212883 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9900000095367432 ) Purity/4FRt1SRqqt*FBt/FRt_purity‚0.25 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 particle/accuracy_3_of_3‚1 +particle/specificity‚0.9435105323791504 7validation_average_jet_accuracy‚0.39420419931411743 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 ."Purity/0FRt0SRqqt3FBt/event_purity‚0.21875 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.3333333432674408 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.5714285969734192 8 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.35264551639556885 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9673131704330444 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9807865023612976 ("Purity/*FRt3SRqqt0FBt/event_purity‚0 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5159050226211548 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.3333333432674408 7 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.2380952537059784 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.9259259700775146 ("Purity/*FRt2SRqqt3FBt/SRqqt_purity‚1 :"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.39941227436065674 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9855635166168213 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9663193821907043 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 8 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.24166664481163025 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9977704882621765 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.03779761865735054 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 *jet/accuracy_1_of_6‚0.9615384340286255 jet/accuracy_6_of_6‚0 jet/accuracy_4_of_7‚0 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9806854128837585 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.0833333358168602 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 -"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.3125 :"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.33927226066589355 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.07589773088693619 7 Purity/2FRt2SRqqt*FBt/FBt_purity‚0.9767441749572754 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.0833333358168602 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 :"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.42994818091392517 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.4344833195209503 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 8 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.36951756477355957 particle/accuracy_1_of_1‚1 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 particle/accuracy_2_of_3‚1 particle/accuracy_3_of_5‚1 , Purity/1FRt3SRqqt1FBt/FRt_purity‚0.28125 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2084587961435318 :"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.40234848856925964 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.4377758502960205 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 particle/accuracy_4_of_4‚1 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.11437702924013138 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 ( Purity/1FRt*SRqqt4FBt/FRt_purity‚0.5 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 & Purity/1FRt4SRqqt*FBt/FRt_purity‚0 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.1507936418056488 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.3918434679508209 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.18967093527317047 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9978017807006836 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 particle/accuracy_1_of_2‚1 jet/accuracy_4_of_6‚0 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9794563055038452 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9701754450798035 ;"Purity/0FRt2SRqqt1FBt/event_purity‚0.004716981202363968 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.4127259850502014 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.6818181872367859 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9137116074562073 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.12949486076831818 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9905590415000916 ;"Purity/1FRt2SRqqt*FBt/event_purity‚0.000933706818614155 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.016288738697767258 <"Purity/1FRt2SRqqt0FBt/event_purity‚0.0014430014416575432 7 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.1916666477918625 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 jet/accuracy_4_of_5‚0 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 8 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.44736841320991516 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.11735910922288895 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.3611111044883728 % _timestamp‚1.7885118017409205e+09 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.46242937445640564 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.4349154531955719 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 8"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.383060485124588 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 *"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.5 & Purity/1FRt3SRqqt2FBt/FRt_purity‚1 particle/accuracy_5_of_7‚1 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 6 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.125339075922966 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9761093258857727 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.3673850893974304 +particle/sensitivity‚0.2646414637565613 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.3874887526035309 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.3660130500793457 & Purity/1FRt0SRqqt4FBt/FRt_purity‚0 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.46931445598602295 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.3263888955116272 7 Purity/0FRt3SRqqt1FBt/FBt_purity‚0.8888888955116272 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.4229815900325775 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 particle/accuracy_4_of_6‚1 & Purity/*FRt3SRqqt2FBt/FRt_purity‚1 ;"Purity/1FRt1SRqqt2FBt/event_purity‚0.010416666977107525 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.10294999927282333 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 ) Purity/3FRt1SRqqt2FBt/FRt_purity‚0.25 ) Purity/4FRt1SRqqt0FBt/FRt_purity‚0.25ûf]Ñk:Û Ŧ 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.3741109371185303 particle/accuracy_2_of_5‚1 jet/accuracy_7_of_7‚0 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_2_of_4‚1 *"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.5 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9837027192115784 :"Purity/*FRt1SRqqt2FBt/event_purity‚0.03353174775838852 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.4264393150806427 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.1277281492948532 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9259259700775146 ( Purity/1FRt0SRqqt3FBt/FRt_purity‚0.5 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.6666666865348816 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 :"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.37385621666908264 * Purity/4FRt*SRqqt*FBt/FRt_purity‚0.125 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 ("Purity/0FRt2SRqqt3FBt/SRqqt_purity‚1 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6236502528190613 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.3214111626148224 (particle/accuracy‚0.8375029563903809 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 ("Purity/4FRt*SRqqt*FBt/SRqqt_purity‚1 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚1 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 -"Purity/1FRt0SRqqt3FBt/event_purity‚0.0625 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.18850575387477875 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9937499761581421 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 8 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.35170507431030273 ;"Purity/*FRt2SRqqt0FBt/event_purity‚0.018949981778860092 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 ( Purity/*FRt*SRqqt4FBt/FRt_purity‚0.5 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.3769841492176056 *jet/accuracy_1_of_3‚0.8083683252334595 +jet/accuracy_3_of_6‚0.03846153989434242 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.13219207525253296 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 & Purity/*FRt1SRqqt4FBt/FRt_purity‚1 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.0623522587120533 7 Purity/*FRt3SRqqt1FBt/FBt_purity‚0.9629629850387573 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.4416269063949585 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 jet/accuracy_5_of_6‚0 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9856921434402466 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6466619372367859 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.1380334496498108 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 _step‚98 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.13026821613311768 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 6 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.208333358168602 *"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.5 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.18227867782115936 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 :"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.43209877610206604 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 jet/accuracy_1_of_7‚1 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9976190328598022 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.1850256621837616 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.21549750864505768 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.20000000298023224 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 8"Purity/1FRt*SRqqt2FBt/event_purity‚0.054151501506567 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.11735910922288895 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.3020833432674408 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 particle/accuracy_1_of_3‚1 particle/accuracy_1_of_5‚1 7 Purity/*FRt*SRqqt4FBt/FBt_purity‚0.8333333134651184 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.4639628529548645 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9907218813896179 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 ("Purity/*FRt3SRqqt*FBt/event_purity‚0 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.1269349753856659 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 +"Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.25 particle/accuracy_6_of_6‚1 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.1870577037334442 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.015942851081490517 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.4298752546310425 particle/accuracy_3_of_6‚1 jet/accuracy_3_of_7‚0 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 +jet/accuracy_2_of_6‚0.32692307233810425 jet/accuracy_5_of_7‚0 :"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.40691903233528137 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.34062495827674866 :"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.41714853048324585 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.7222222089767456 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.4166666567325592 +"Purity/1FRt1SRqqt3FBt/SRqqt_purity‚0.75 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.4444444477558136 7 Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9166666865348816 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 7 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.9047619104385376 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 8 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.47931405901908875 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 :"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.43736281991004944 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 :"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.43965521454811096 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 7 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.1796296387910843 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 :"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.34185802936553955 :"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.31536099314689636 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.12314967811107635 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.35227179527282715 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 9"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.2995210587978363 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.1528686136007309 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9166667461395264 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 ;"Purity/0FRt2SRqqt*FBt/event_purity‚0.036365997046232224 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2916666567325592 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 * Purity/*FRt2SRqqt2FBt/FRt_purity‚0.375 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9687660336494446 6 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.523217499256134 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.004904275760054588 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9950029850006104 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 ;"Purity/0FRt1SRqqt2FBt/event_purity‚0.056342363357543945 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.18391813337802887 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.9166666865348816 :"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.36364561319351196 :"Purity/0FRt*SRqqt2FBt/event_purity‚0.22959132492542267 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.0833333358168602 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.04370106756687164 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.33812612295150757 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.4314594268798828 'particle/f_score‚0.3370826840400696 7 Purity/*FRt3SRqqt*FBt/FBt_purity‚0.9642857313156128 particle/accuracy_7_of_7‚1 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.19171233475208282 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 8"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.315907746553421 :"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.42424097657203674 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.4483811557292938 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9659090638160706 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 particle/accuracy_6_of_7‚1 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.4490540027618408 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.0555555559694767 +"Purity/1FRt*SRqqt3FBt/SRqqt_purity‚0.75 * Purity/4FRt*SRqqt0FBt/FRt_purity‚0.125 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.2150954306125641 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.18999351561069489 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.3166666626930237 +"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.75 *"Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0.5 :"Purity/0FRt*SRqqt0FBt/event_purity‚0.34765133261680603 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.4377758502960205 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.012812395580112934 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 8"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.474702388048172 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.09917915612459183 7 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.1842963546514511 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 *jet/accuracy_2_of_4‚0.2926087975502014 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.05519581213593483 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5169184803962708 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.10288066416978836 jet/accuracy_5_of_5‚0 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.13111110031604767 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9766128063201904 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.37950366735458374 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.5326389670372009 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.1904793530702591 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9924511313438416 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.09KGôáX87723246216774 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 <"Purity/2FRt*SRqqt0FBt/event_purity‚0.0015432098880410194 *"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.5 jet/accuracy_6_of_7‚0 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.33947011828422546 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.2666666805744171 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.4080445170402527 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.4629371464252472 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.3577527105808258 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 *jet/accuracy_2_of_3‚0.1956164389848709 - Purity/0FRt0SRqqt3FBt/FBt_purity‚0.921875 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 & Purity/1FRt2SRqqt2FBt/FBt_purity‚1 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 *jet/accuracy_1_of_2‚0.6960355043411255 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.9078947305679321 ;"Purity/1FRt1SRqqt1FBt/event_purity‚0.015317751094698906 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 8 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.32592591643333435 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.4166666567325592 ("Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.4094215929508209 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.20000000298023224 ("Purity/0FRt3SRqqt0FBt/event_purity‚0 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.15946808457374573 <"Purity/*FRt2SRqqt1FBt/event_purity‚0.0022675737272948027 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.10862766206264496 particle/accuracy_3_of_7‚1 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.08206985145807266 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 particle/accuracy_5_of_6‚1 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9864341616630554 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.00515872985124588 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 *jet/accuracy_1_of_1‚0.4453316330909729 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.29794153571128845 7 Purity/2FRt*SRqqt1FBt/FBt_purity‚0.9960317611694336 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 :"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.29629626870155334 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 +"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.75 7 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.1533239781856537 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 particle/accuracy_5_of_5‚1 6 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.357394278049469 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.06818182021379471 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 :"Purity/0FRt0SRqqt2FBt/event_purity‚0.29543691873550415 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.4770068824291229 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 )jet/accuracy_3_of_4‚0.035651795566082 *jet/accuracy_1_of_5‚0.9234639406204224 9"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.3499999940395355 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.18291912972927094 :"Purity/2FRt0SRqqt2FBt/event_purity‚0.02631578966975212 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 ("Purity/0FRt3SRqqt*FBt/event_purity‚0 ( Purity/1FRt1SRqqt3FBt/FRt_purity‚0.5 _runtime‚1427.216771958 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚1 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 ( Purity/*FRt0SRqqt3FBt/FRt_purity‚0.5 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.12614542245864868 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.2305431067943573 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9965516924858093 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.0833333358168602 jet/accuracy_2_of_7‚0.5 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9737676382064819 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9809044003486633 particle/accuracy_1_of_4‚1 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.1666666716337204 particle/accuracy_2_of_7‚1 7 Purity/0FRt2SRqqt1FBt/FBt_purity‚0.9773584604263306 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.3747023940086365 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 trainer/global_step‚1208 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9868219494819641 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8032798767089844 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 9"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.3333333432674408 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 particle/accuracy_4_of_7‚1 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 particle/accuracy_2_of_6‚1 ("Purity/4FRt*SRqqt0FBt/SRqqt_purity‚1 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.3324505388736725 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.4166666567325592 :"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.36214855313301086 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9977704882621765 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.5222986340522766 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.0833333358168602 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.34009239077568054 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 & Purity/*FRt0SRqqt4FBt/FRt_purity‚0 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.4631842076778412 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 particle/accuracy_3_of_4‚1 + Purity/0FRt*SRqqt4FBt/FBt_purity‚0.8125 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 8 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.38081395626068115 6 Purity/2FRt2SRqqt1FBt/FBt_purity‚0.976190447807312 8 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.16981980204582214 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 +jet/accuracy_3_of_3‚0.01686221733689308 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.19672457873821259 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.012913917191326618 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.014109919779002666 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 & Purity/1FRt1SRqqt4FBt/FRt_purity‚1 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.11528763920068741 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 :"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.30454543232917786 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 particle/accuracy_4_of_5‚1 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9857938885688782 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 -jet/accuracy_4_of_4‚0.0009920635493472219 particle/accuracy_2_of_2‚1 particle/accuracy_1_of_6‚1 7 Purity/0FRt2SRqqt2FBt/FBt_purity‚0.9642857313156128 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9948357939720154 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.14059039950370789 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 8 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.36666667461395264 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.15158100426197052 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 +"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.25 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 & Purity/1FRt4SRqqt0FBt/FRt_purity‚0 *loss/val_total_loss‚50.640127528021544 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.0715567097067833 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 6 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.991266131401062 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 7 Purity/1FRt0SRqqt1FBt/FBt_purity‚0.9980308413505554 :"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.35905328392982483 8 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.43464696407318115 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9773243069648743 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9845403432846069 6 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.344946950674057 :"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.36666667461395264 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.27327588200569153 9"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.2927875220775604 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2154974788427353 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.15559065341949463 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 ;"Purity/2FRt*SRqqt2FBt/event_purity‚0.023255813866853714 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5327509045600891 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 *jet/accuracy_1_of_4‚0.8691949844360352 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.5263158082962036 + Purity/0FRt0SRqqt4FBt/FBt_purity‚0.8125 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.14590950310230255 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.2857142984867096 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 & Purity/3FRt4SRqqt0FBt/FRt_purity‚0 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 +"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.75 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.13475176692008972 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.4044511020183563 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8500000238418579 particle/accuracy_1_of_7‚1 7 Purity/0FRt3SRqqt*FBt/FBt_purity‚0.8888888955116272 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 8"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.286636620759964 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 :"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.26543211936950684 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 8 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.36742427945137024 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.12500807642936707 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 6 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.167772576212883 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9900000095367432 ) Purity/4FRt1SRqqt*FBt/FRt_purity‚0.25 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 particle/accuracy_3_of_3‚1 +particle/specificity‚0.9435105323791504 7validation_average_jet_accuracy‚0.39420419931411743 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 ."Purity/0FRt0SRqqt3FBt/event_purity‚0.21875 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.3333333432674408 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.5714285969734192 8 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.35264551639556885 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9673131704330444 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9807865023612976 ("Purity/*FRt3SRqqt0FBt/event_purity‚0 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5159050226211548 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.3333333432674408 7 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.2380952537059784 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.9259259700775146 ("Purity/*FRt2SRqqt3FBt/SRqqt_purity‚1 :"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.39941227436065674 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9855635166168213 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9663193821907043 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 8 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.24166664481163025 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9977704882621765 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.03779761865735054 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 *jet/accuracy_1_of_6‚0.9615384340286255 jet/accuracy_6_of_6‚0 jet/accuracy_4_of_7‚0 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9806854128837585 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.0833333358168602 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 -"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.3125 :"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.33927226066589355 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.07589773088693619 7 Purity/2FRt2SRqqt*FBt/FBt_purity‚0.9767441749572754 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.0833333358168602 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 :"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.42994818091392517 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.4344833195209503 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 8 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.36951756477355957 particle/accuracy_1_of_1‚1 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 particle/accuracy_2_of_3‚1 particle/accuracy_3_of_5‚1 , Purity/1FRt3SRqqt1FBt/FRt_purity‚0.28125 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2084587961435318 :"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.40234848856925964 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.4377758502960205 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 particle/accuracy_4_of_4‚1 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.11437702924013138 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 ( Purity/1FRt*SRqqt4FBt/FRt_purity‚0.5 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 & Purity/1FRt4SRqqt*FBt/FRt_purity‚0 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.1507936418056488 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.3918434679508209 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.18967093527317047 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9978017807006836 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 particle/accuracy_1_of_2‚1 jet/accuracy_4_of_6‚0 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9794563055038452 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9701754450798035 ;"Purity/0FRt2SRqqt1FBt/event_purity‚0.004716981202363968 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.4127259850502014 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.6818181872367859 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9137116074562073 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.12949486076831818 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9905590415000916 ;"Purity/1FRt2SRqqt*FBt/event_purity‚0.000933706818614155 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.016288738697767258 <"Purity/1FRt2SRqqt0FBt/event_purity‚0.0014430014416575432 7 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.1916666477918625 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 jet/accuracy_4_of_5‚0 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 8 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.44736841320991516 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.11735910922288895 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.3611111044883728 % _timestamp‚1.7885118017409205e+09 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.46242937445640564 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.4349154531955719 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 8"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.383060485124588 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 *"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.5 & Purity/1FRt3SRqqt2FBt/FRt_purity‚1 particle/accuracy_5_of_7‚1 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 6 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.125339075922966 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9761093258857727 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.3673850893974304 +particle/sensitivity‚0.2646414637565613 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.3874887526035309 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.3660130500793457 & Purity/1FRt0SRqqt4FBt/FRt_purity‚0 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.46931445598602295 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.3263888955116272 7 Purity/0FRt3SRqqt1FBt/FBt_purity‚0.8888888955116272 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.4229815900325775 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 particle/accuracy_4_of_6‚1 & Purity/*FRt3SRqqt2FBt/FRt_purity‚1 ;"Purity/1FRt1SRqqt2FBt/event_purity‚0.010416666977107525 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.10294999927282333 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 ) Purity/3FRt1SRqqt2FBt/FRt_purity‚0.25 ) Purity/4FRt1SRqqt0FBt/FRt_purity‚0.25 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.18421052396297455 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.3571428656578064 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 +jet/accuracy_2_of_5‚0.37919867038726807 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 ,"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.625 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 epoch‚0 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9879372715950012 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9944315552711487 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 +jet/accuracy_3_of_5‚0.07282385975122452 +"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.25 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 & Purity/3FRt4SRqqt*FBt/FRt_purity‚0 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.3177083432674408 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 +jet/accuracy_2_of_2‚0.07419481873512268 ("Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0 ( Purity/*FRt1SRqqt3FBt/FRt_purity‚0.5 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9845969676971436 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9868807792663574 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 :"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.43886834383010864bn#MÆJÜ ‚(:¿ ÆŒêÔÈ”üÿ disk.sda.out‚ 163.484375 disk.sda2.in‚ 0.0390625 disk.sda2.out‚ 163.484375 disk.nvme1n1.in‚ 0.05078125! disk.nvme1n1.out‚ 6827.1328125 disk.md0p1.in‚ 0.0703125# memory_percent‚92.0320987701416 disk.sdb.out‚ 163.484375 disk.md0p1.out‚ 157.64453125( proc.memory.availableMB‚ 1305.4609375" disk.nvme0n1.out‚ 8018.23828125 disk.rbd0.out‚ 0.279296875 disk.sdb1.out‚0 disk.dm-0.out‚14845.37109375 proc.cpu.threads‚17 disk.sda1.out‚0 network.sent‚ 1.813624e+06 disk.md0.in‚ 0.0703125 disk.rbd2.in‚0 network.recv‚524366% disk./.usageGB‚2711.4137420654297 disk.nvme0n1.in‚ 0.02734375 disk.sdb1.in‚0 disk.rbd2.out‚ 0.3779296875 cpu‚25.243334268699872 disk.sdb2.in‚0.03125 disk.sda1.in‚0 disk.dm-0.in‚0.078125) disk./.usagePercent‚75.81226304296183 disk.sdb.in‚0.03125 disk.sda.in‚ 0.0390625 disk.rbd0.in‚0 disk.md0.out‚ 157.64453125 proc.memory.rssMB‚ 2996.40625* proc.memory.percent‚18.288612365722656 disk.sdb2.out‚ 163.484375h@Ó}Ý ‚(:’ ÆŒêÔêûÓ€ gpu.0.gpu‚52.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚36.0 gpu.0.powerWatts‚117.81' gpu.0.enforcedPowerLimitWatts‚400.0) gpu.0.powerPercent‚29.452499999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¿Ûþ!’Þ ‚(:‡ ÕŒêÔŠ‚ðÿ gpu.0.gpu‚47.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚165.93' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚41.4825 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0p¬[ß ‚(:Ð ÕŒêÔç⨀) disk./.usagePercent‚75.81225664332992 disk.rbd2.out‚ 0.44091796875 disk.sdb2.in‚0.03125 disk.dm-0.in‚0.078125 disk.sda.out‚ 165.447265625 disk.md0.in‚ 0.0703125 disk.md0p1.out‚ 159.53515625 network.recv‚528978" disk.nvme0n1.out‚ 8018.23828125 disk.sdb.in‚0.03125 disk.sdb.out‚ 165.447265625! disk.nvme1n1.out‚ 6875.6484375 disk.md0.out‚ 159.53515625 disk.sda1.out‚0 disk.sda1.in‚0 disk.nvme0n1.in‚ 0.02734375 disk.sda2.in‚ 0.0390625 disk.sda2.out‚ 165.447265625 disk.nvme1n1.in‚ 0.05078125 disk.sdb2.out‚ 165.447265625 disk.md0p1.in‚ 0.0703125# proc.memory.rssMB‚ 2996.41796875 network.sent‚ 1.925491e+06 disk.sdb1.in‚0) proc.memory.availableMB‚ 1305.29296875 disk.rbd0.out‚ 0.279296875 disk.rbd2.in‚0$ memory_percent‚92.03312397003174 cpu‚25.24119400727114 proc.cpu.threads‚17 disk.dm-0.out‚14893.88671875% disk./.usageGB‚2711.4135131835938* proc.memory.percent‚18.288683891296387 disk.sda.in‚ 0.0390625 disk.rbd0.in‚0 disk.sdb1.out‚0@ þå”à ‚(:‰ äŒêÔê…¹ÿ gpu.0.gpu‚40.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚173.059' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.26475 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0øícTá ‚(:É äŒêÔø‚€ network.recv‚533426% disk./.usageGB‚2711.3610801696777 disk.sda2.in‚ 0.0390625 disk.sda2.out‚ 166.99609375 disk.dm-0.in‚0.078125 proc.cpu.threads‚17 disk.rbd0.in‚0 disk.sdb1.out‚0 disk.sdb2.in‚0.03125# proc.memory.rssMB‚ 2996.41796875 disk.nvme1n1.in‚ 0.05078125 network.sent‚ 1.929332e+06 disk.sdb2.out‚ 166.99609375 disk.sdb.out‚ 166.99609375* proc.memory.percent‚18.288683891296387 disk.rbd0.out‚ 0.279296875 disk.rbd2.out‚ 0.44091796875 disk.md0p1.in‚ 0.0703125 cpu‚25.23894944314958" disk.nvme0n1.out‚ 8667.19921875( disk./.usagePercent‚75.8107905943159 disk.sdb1.in‚0 disk.md0.out‚ 161.01171875 disk.rbd2.in‚0 disk.nvme0n1.in‚ 0.02734375 disk.md0.in‚ 0.0703125 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-0.out‚ 15655.734375 disk.md0p1.out‚ 161.01171875 disk.sda.in‚ 0.0390625 disk.sda.out‚ 166.99609375) proc.memory.availableMB‚ 1305.07421875" disk.nvme1n1.out‚ 6988.53515625 disk.sdb.in‚0.03125# memory_percent‚92.0344591140747Ð2k@2â - _wandbruntime‚1479  _runtime‚1479øcæÅã ¿ _step‚99 & lr-AdamW/pg1‚0.0008576720012635259 & lr-AdamW/pg2‚0.0008576720012635259 trainer/global_step‚1249 $ _timestamp‚1.788511854106612e+09 _runtime‚1479.577274555ÐÇ¿ÂÉä à & lr-AdamW/pg1‚0.0008576720012635259 & lr-AdamW/pg2‚0.0008576720012635259 trainer/global_step‚1249 $ _timestamp‚1.788511854106612e+09 _runtime‚1479.577274555 _step‚99c+óZ2å - _wandbruntime‚1479  _runtime‚1479¡¹N¤æ æO JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚31393055 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚104360983835648 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899406848 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177046016 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚116645229947904 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚102938281657856 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955590144 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1421802770944 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚102938281657856 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26086 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚23610109 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177046016 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚586 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚31393055 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007233 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1421802770944 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7539336 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚102049609347237 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚31366969 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31929 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚25845544 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚25845542 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚102938263654912 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚102938263654912 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899406848 _runtime‚1479.578790431 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 263286784 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 398533632 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26088 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1422702177792 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1166 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚584 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚115162665862144 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917409792 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1191 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1482564085760 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚104360066425856 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚57238599 trainer/global_step‚1249 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚25845544 % _timestamp‚1.7885118541078084e+09 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚25845542 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚16070773 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31929 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411412 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1416501492033 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚104360066425856 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1422702177792 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚23610695 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚115162641922048 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917409792 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚31366969 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31684 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955590144 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26088 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1422702177792 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1166 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚584 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚115162665862144 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917409792 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1191 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1482564085760 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚104360066425856 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚57238599 trainer/global_step‚1249 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚25845544 % _timestamp‚1.7885118541078084e+09 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚25845542 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚16070773 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31929 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411412 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1416501492033 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚104360066425856 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1422702177792 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚23610695 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚115162641922048 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917409792 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚31366969 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31684 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955590144 IDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚23628067 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚31417672 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚16083151 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1166 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚57283114 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1422915574272 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚100710173498560 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚1249 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚31391560 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1191 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _runtime‚1480.767309919 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899420160 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚25865442 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚25865440 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411620 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1417610153351 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚103017893950464 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚115251985082880 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 % _timestamp‚1.7885118552951925e+09 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26114 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899420160 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚116735448530432 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1418507564971 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177046016 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 224653312 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1483487387648 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚101 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚31391560 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1423814994432 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31929 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177046016 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚57283114 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚25865440 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚104441726947840 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚104440809524736 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214657 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚104441726947840 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31929 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚576 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917423104 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955590144 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171©‡öê ôM EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚101 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚31391560 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1423814994432 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31929 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177046016 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚57283114 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚25865440 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚104441726947840 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚104440809524736 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214657 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚104441726947840 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31929 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚576 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917423104 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955590144 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚57257000 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 248593408 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚31417672 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1483712040960 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚57257000 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450980 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚23628067 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚31417672 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚16083151 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1166 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚57283114 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1422915574272 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚100710173498560 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚1249 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚31391560 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1191 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _runtime‚1480.767309919 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899420160 YIDeviceStatsMonitor.on_train_batch_end/¡‹æœ reserved_bytes.large_pool.allocated‚ 13037993984 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚25865442 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚25865440 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411620 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1417610153351 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚103017893950464 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚115251985082880 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 % _timestamp‚1.7885118552951925e+09 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26114 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899420160 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚116735448530432 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1418507564971 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177046016 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 224653312 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1483487387648 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 FDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1540357624832 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214657 _step‚104 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚59440313 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1478356758016 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚26840442 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899406848 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31929 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1477457351168 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚106004060710745 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚16689575 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚121168731722752 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26088 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1166 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚104531215150192 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚119628602958848 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚108397235783680 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177046016 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚106919796435456 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚104531198110832 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚24514653 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚16689573 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 252801024 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 $ _timestamp‚1.788511909951651e+09 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7824461 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31684 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚121168984523776 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 398533632 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31684 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177046016 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1191 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450772 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚32625957 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚106003146259973 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚106919778432512 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955590144 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1471948149141 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1478356758016 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1477457351168 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚24514034 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 trainer/global_step‚1299 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚26840442 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26086 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7825078 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26088 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1166 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚104531215150192 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚119628602958848 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚108397235783680 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177046016 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚106919796435456 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚104531198110832 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚24514653 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚16689573 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 252801024 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 $ _timestamp‚1.788511909951651e+09 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7824461 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31684 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚121168984523776 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 398533632 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31684 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177046016 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1191 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450772 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚32625957 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚106003146259973 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚106919778432512 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955590144 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1471948149141 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1478356758016 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1477457351168 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚24514034 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 trainer/global_step‚1299 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚26840442 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26086 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7825078 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1540357624832 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214657 _step‚104 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚59440313 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1478356758016 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚26840442 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899406848 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31929 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1477457351168 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚106004060710745hÌÔ­Ú%ü - _wandbruntime‚1536  _r†2ˆ untime‚1536Y”C®ö&ý ðM JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚106999408728064 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚24532017 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚121259454831616 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 trainer/global_step‚1299 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚24532628 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31929 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1166 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177046016 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚32650574 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1478570154496 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚106999408728064 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚121259193655296 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450980 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 398533632 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917423104 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31929 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚108478896305664 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899420160 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚16701949 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _runtime‚1536.513151783 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26112 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚26860342 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚26860342 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚108478896305664 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 237236224 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚106083149466399 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1473954222079 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚59510916 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚59510916 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚59484802 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚16701951 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚106082235015419 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚59484802 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 _step‚105 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚104609195244320 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177046016 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1541508711936 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚108477978882560 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955590144 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917423104 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚32650574 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7830068 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1541271475712 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1473056810459 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26112 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚32624462 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26114 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚609 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214657 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177046016 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚32650574 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1478570154496 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚106999408728064 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚121259193655296 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450980 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 398533632 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917423104 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31929 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚108478896305664 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899420160 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚16701949 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _runtime‚1536.513151783 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26112 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚26860342 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚26860342 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚108478896305664 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 237236224 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚106083149466399 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1473954222079 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚59510916 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚59510916 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚59484802 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚16701951 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚106082235015419 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚59484802 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 _step‚105 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚104609195244320 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177046016 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1541508711936 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚108477978882560 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955590144 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917423104 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚32650574 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7830068 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1541271475712 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1473056810459 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26112 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚32624462 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26114 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚609 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214657 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚24532017 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚121259454831616 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 trainer/global_step‚1299 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚24532628 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31929 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1166 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26088 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚112434405141504 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 235152384 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚124094587935744 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450772 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚27835342 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8110400 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31684 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚112435322551296 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚109957597623481 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26088 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917409792 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1534011338240 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚110901293210112 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚33858859 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚630 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚61694203 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚27835344 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1528292217661 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917409792 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8111030 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 trainer/global_step‚1349 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1527394806249 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0ÍŠð'êO H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31684 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚112435322551296 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚109957597623481 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26088 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917409792 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1534011338240 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚110901293210112 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚33858859 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚630 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚61694203 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚27835344 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1528292217661 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917409792 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8111030 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 trainer/global_step‚1349 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1527394806249 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚27835344 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1533111931392 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411412 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26088 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚112434405141504 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 235152384 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚124094587935744 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450772 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚27835342 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8110400 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171loÈÝ2Ž- _wandbruntime‚1588  _runtime‚1588Wò¿.ö&ðM PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 _step‚109 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚110980923505664 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚25437448 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8116697 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚108508199950720 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1529400879187 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚33857364 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31929 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26112 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214657 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚25436847 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 _runtime‚1588.431175096 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1534224734720 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚61712604 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917423104 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚61738718 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚110036686378927 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚124183907156480 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26114 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚110980941508608 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899420160 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1528503467567 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚61712604 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1534224734720 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚125783371412992 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚27855242 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚125783122819584 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31929 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177046016 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚124183883216384 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚27855242 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177046016 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚599 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1535124154880 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚33883476 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚33857364 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1191 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007233 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411620 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1535124154880 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚17320751 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 trainer/global_step‚1349 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955590144 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚17320749 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 % _timestamp‚1.7885119629603117e+09 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917423104 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31684 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564G”`áÉôM NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚33883476 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26112 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 248593408 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚25436847 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 _runtime‚1588.431175096 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1534224734720 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚61712604 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917423104 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚61738718 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚110036686378927 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚124183907156480 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26114 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚110980941508608 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899420160 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1528503467567 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚61712604 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1534224734720 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚125783371412992 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚27855242 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚125783122819584 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31929 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177046016 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚124183883216384 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚27855242 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177046016 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚599 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1535124154880 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚33883476 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚33857364 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1191 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007233 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411620 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1535124154880 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚17320751 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 trainer/global_step‚1349 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955590144 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚17320749 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 % _timestamp‚1.7885119629603117e+09 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917423104 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31684 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 _step‚109 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚110980923505664 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚25437448 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8116697 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚108508199950720 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1529400879187 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚33857364 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31929 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26112 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214657 EDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177046016 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 _step‚112 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚116471574499328 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899406848 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26086 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1589665918464 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚128560548972544 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1656256540160 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚113912048986989 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007233 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1191 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚35065675 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899406848 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26086 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31929 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31684 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚28830242 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚621 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917409792 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚26325542 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26088 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚35065675 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 398533632 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214657 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26088 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917409792 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1166 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚130216805512704 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31684 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955590144 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31929 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1582841463357 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚619 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1583738874769 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955590144 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚114882807987712 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚17927175 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1589665918464 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 254898176 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450772 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚63895917 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚114882807987712 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177046016 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411412 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚128560525032448 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚112329224562992 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚28830242 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚35091761 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8397748–™¥Çî'¡éO R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚116471574499328 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899406848 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26086 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1589665918464 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚128560548972544 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1656256540160 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚113912048986989 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007233 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1191 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚35065675 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899406848 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26086 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31929 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31684 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚28830242 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚621 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917409792 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚26325542 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26088 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚35065675 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 398533632 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214657 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26088 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917409792 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1166 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚130216805512704 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31684 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955590144 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31929 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1582841463357 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚619 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1583738874769 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955590144 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚114882807987712 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚17927175 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1589665918464 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 254898176 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450772 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚63895917 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚114882807987712 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177046016 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411412 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚128560525032448 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚112329224562992 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚28830242 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚35091761 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8397748 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 230958080 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚116472491909120 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚26324921 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚113912963437761 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177046016 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 _step‚112Ǧ°upmùòÃ2¢- _wandbruntime‚1643  _runtime‚1643jõÔUö&£ðM E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚112407187617760 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411620 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007233 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚63966520 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚35090266 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚26342948 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1166 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚130307273606144 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 398533632 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 230944768 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899420160 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1657405412864 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚116553235021312 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917423104 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚28850140 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214657 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚112407204657120 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26112 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917423104 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚28850142 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚28850140 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚114962456286208 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚130307018721280 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1584847536295 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177046016 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚114962438283264 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚116552317598208 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8403399 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚28850142 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955590144 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚113992052193415 _step‚113 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚114962456286208 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚1643.154336219 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1589879314944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31929 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚608 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1590778735104 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1191 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8404005 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚113991137742435 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 trainer/global_step‚1399 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚17939549 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚116553235021312 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955590144 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚35090266 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 254884864 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚128649868193280 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚17939551 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚114962438283264 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚606 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚63940406 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚63940406 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1657174468096 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31929 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚63966520 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450980 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0s5|ðú&¤ôM ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31929 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚63966520 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450980 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚112407187617760 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411620 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007233 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚63966520 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚35090266 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚26342948 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1166 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚130307273606144 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 398533632 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 230944768 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899420160 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1657405412864 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚116553235021312 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917423104 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚28850140 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214657 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚112407204657120 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26112 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917423104 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚28850142 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚28850140 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚114962456286208 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚130307018721280 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1584847536295 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177046016 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚114962438283264 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚116552317598208 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8403399 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚28850142 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955590144 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚113992052193415 _step‚113 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚114962456286208 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚1643.154336219 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1589879314944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31929 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚608 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1590778735104 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1191 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8404005 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚113991137742435 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 trainer/global_step‚1399 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚17939549 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚116553235021312 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955590144 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚35090266 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 254884864 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚128649868193280 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚17939551 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚114962438283264 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚606 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚63940406 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚63940406 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1657174468096q–8˜Ù2¥- _wandbruntime‚1643  _runtime‚1643 ^^*\¦Ö 0loss/FRt4/assignment_loss‚4.5333689749240875 1loss/SRqqt4/assignment_loss‚2.035612933775958 1loss/SRqqt1/detection_loss‚2.0696712475876478 /loss/FBt4/detection_loss‚2.5677367874554227 /loss/FRt2/detection_loss‚1.5987095329220649 0loss/FBt4/assignment_loss‚0.4227810853293964 .loss/FBt2/detection_loss‚0.710325218963986 % _timestamp‚1.7885120176857982e+09 /loss/FRt1/detection_loss‚2.2636759700001896 .loss/FRt4/detection_loss‚2.896741190126964 .loss/FBt1/detection_loss‚2.320023444082056 &loss/total_loss‚46.424725525644774 trainer/global_step‚1399 _step‚114 0loss/FRt2/assignment_loss‚1.6118105318811204 2loss/SRqqt2/assignment_loss‚1.2355829877647715 0loss/FBt2/assignment_loss‚0.2986817825097337 epoch‚1 /loss/FBt1/assignment_loss‚0.697091842336314 1loss/SRqqt3/detection_loss‚1.6723881212491838 0loss/FRt1/assignment_loss‚3.9391349779592977 2loss/SRqqt3/assignment_loss‚1.4951221713766691 1loss/SRqqt2/detection_loss‚1.0165000037408212 0loss/SRqqt4/detection_loss‚2.331275627017021 _runtime‚1643.156029527 2loss/SRqqt1/assignment_loss‚1.7242688460405482 /loss/FRt3/detection_loss‚2.2325514166251472 0loss/FRt3/assignment_loss‚2.6032464245091314 2loss/FBt3/assignment_loss‚0.014752329016725222 /loss/FBt3/detection_loss‚4.1336720784505205E,t`§Ú 0loss/FRt3/assignment_loss‚2.6032464245091314 2loss/FBt3/assignment_loss‚0.014752329016725222 /loss/FBt3/detection_loss‚4.1336720784505205 0loss/FRt4/assignment_loss‚4.5333689749240875 1loss/SRqqt4/assignment_loss‚2.035612933775958 1loss/SRqqt1/detection_loss‚2.0696712475876478 /loss/FBt4/detection_loss‚2.5677367874554227 /loss/FRt2/detection_loss‚1.5987095329220649 0loss/FBt4/assignment_loss‚0.4227810853293964 .loss/FBt2/detection_loss‚0.710325218963986 % _timestamp‚1.7885120176857982e+09 /loss/FRt1/detection_loss‚2.2636759700001896 .loss/FRt4/detection_loss‚2.896741190126964 .loss/FBt1/detection_loss‚2.320023444082056 &loss/total_loss‚46.424725525644774 trainer/global_step‚1399 _step‚114 0loss/FRt2/assignment_loss‚1.6118105318811204 2loss/SRqqt2/assignment_loss‚1.2355829877647715 0loss/FBt2/assignment_loss‚0.2986817825097337 epoch‚1 /loss/FBt1/assignment_loss‚0.697091842336314 1loss/SRqqt3/detection_loss‚1.6723881212491838 0loss/FRt1/assignment_loss‚3.9391349779592977 2loss/SRqqt3/assignment_loss‚1.4951221713766691 1loss/SRqqt2/detection_loss‚1.0165000037408212 0loss/SRqqt4/detection_loss‚2.331275627017021 _runtime‚1643.156029527 2loss/SRqqt1/assignment_loss‚1.7242688460405482 /loss/FRt3/detection_loss‚2.2325514166251472r7‹“¨‚(:ˆ ˜ŽêÔ³÷ùþ gpu.0.gpu‚48.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚37.0 gpu.0.powerWatts‚118.142' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚29.5355 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‘ÃÓÞG©‚(:¼ ˜ŽêÔ³ª«ÿ network.recv‚587253 disk.md0p1.out‚ 180.65234375 disk.nvme0n1.in‚ 0.02734375 disk.sda.out‚ 187.41015625$ memory_percent‚99.96531009674072 disk.md0.out‚ 180.65234375 disk.md0p1.in‚ 0.0703125& proc.memory.availableMB‚ 5.68359375) disk./.usagePercent‚75.81997609268169$ disk./.usageGB‚2711.689598083496 disk.rbd2.in‚0 disk.sda1.out‚0 disk.sda.in‚ 0.0390625* proc.memory.percent‚18.288707733154297 network.sent‚ 2.304722e+06! disk.nvme0n1.out‚ 10132.765625 disk.sdb2.out‚ 187.40625 disk.sdb.out‚ 187.40625 disk.rbd0.in‚0 cpu‚25.215092474022203 disk.sda2.out‚ 187.41015625 disk.md0.in‚ 0.0703125 disk.sdb2.in‚0.03125 disk.sda1.in‚0 disk.dm-0.in‚0.078125 disk.dm-0.out‚ 17797.0234375 disk.sda2.in‚ 0.0390625 disk.rbd0.out‚ 0.279296875 disk.nvme1n1.in‚ 0.05078125 disk.sdb1.out‚0 disk.rbd2.out‚ 0.50390625 proc.cpu.threads‚17! disk.nvme1n1.out‚ 7664.2578125 disk.sdb1.in‚0 disk.sdb.in‚0.03125! proc.memory.rssMB‚ 2996.421875¹Ø¤·’ª‚(:‡ §ŽêÔꌀþ gpu.0.gpu‚38.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚160.028' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚40.007 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0PÀ¿üQ«‚(:Æ §ŽêÔ«©ÿÿ disk.dm-0.out‚17945.74609375 disk.md0p1.out‚ 182.703125 disk.sdb.in‚0.03125 disk.md0.out‚ 182.703125 disk.rbd2.in‚0! disk.sdb2.out‚189.53076171875 disk.sda1.in‚0 disk.nvme0n1.out‚ 10281.34375 disk.sda.in‚ 0.0390625! proc.memory.rssMB‚ 2996.421875 network.recv‚593129$ disk./.usageGB‚2711.411952972412 disk.sdb1.out‚0 disk.dm-0.in‚0.078125$ memory_percent‚99.96471405029297* proc.memory.percent‚18.288707733154297 disk.sdb1.in‚0 disk.sdb2.in‚0.03125 disk.sda.out‚189.53076171875# proc.memory.availableMB‚5.78125 network.sent‚ 2.41443e+06 disk.sda1.out‚0 disk.nvme0n1.in‚ 0.02734375 disk.sdb.out‚189.53076171875! disk.sda2.out‚189.53076171875) disk./.usagePercent‚75.81221301917228 disk.rbd2.out‚ 0.50390625 disk.sda2.in‚ 0.0390625 cpu‚25.21345876431361 disk.rbd0.in‚0" disk.nvme1n1.out‚ 7664.40234375 disk.md0p1.in‚ 0.0703125 proc.cpu.threads‚17 disk.rbd0.out‚ 0.279296875 disk.md0.in‚ 0.0703125 disk.nvme1n1.in‚ 0.05078125pГ¬‚(:ˆ ¶ŽêÔÁåôý gpu.0.gpu‚30.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚175.058' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.7645 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0îW~tX­‚(:Í ¶ŽêÔ݃Îÿ disk.sdb.in‚0.03125! disk.sda2.out‚191.31982421875 disk.nvme1n1.in‚ 0.05078125 disk.sdb1.out‚0 disk.sdb2.in‚0.03125 disk.sda1.out‚0 disk.dm-0.out‚ 18046.6484375 disk.rbd0.out‚ 0.279296875& proc.memory.availableMB‚ 5.41015625 network.sent‚ 2.417689e+06 network.recv‚597290) disk./.usagePercent‚75.81081683280681 disk.nvme1n1.out‚ 7664.65625 disk.sdb1.in‚0 disk.sda1.in‚0 disk.dm-0.in‚0.078125$ memory_percent‚99.96697902679443 disk.md0p1.in‚ 0.0703125" disk.nvme0n1.out‚ 10381.9921875 disk.md0.in‚ 0.0703125! disk.sdb2.out‚191.31982421875$ disk./.usageGB‚2711.362018585205 disk.sdb.out‚191.31982421875 disk.sda.out‚191.31982421875 proc.cpu.threads‚17 disk.md0p1.out‚ 184.4140625 disk.nvme0n1.in‚ 0.02734375 disk.sda.in‚ 0.0390625 disk.sda2.in‚ 0.0390625 disk.rbd0.in‚0 disk.rbd2.in‚0 disk.rbd2.out‚ 0.50390625# proc.memory.rssMB‚ 2996.42578125 disk.md0.out‚ 184.4140625* proc.memory.percent‚18.288731575012207 cpu‚25.211887018309895HB@¬“®‚(:ˆ ÅŽêÔƒ¥ÿ gpu.0.gpu‚41.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚165.598' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚41.3995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0+#ÛaY¯‚(:Î ÅŽêÔ¾´üÿ! disk.sda2.out‚192.81103515625 disk.rbd0.in‚0 disk.sdb1.out‚0 disk.md0.out‚ 185.84375 disk.rbd2.out‚ 0.50390625 disk.nvme1n1.in‚ 0.05078125 disk.sdb2.in‚0.03125 disk.dm-0.out‚18212.09765625 disk.md0p1.out‚ 185.84375 proc.cpu.threads‚17 disk.rbd0.out‚ 0.279296875! disk.nvme1n1.out‚ 7742.0078125 disk.md0.in‚ 0.0703125 disk.sda1.out‚0# proc.memory.rssMB‚ 2996.42578125 network.sent‚ 2.425201e+06 disk.nvme0n1.in‚ 0.02734375$ memory_percent‚99.96535778045654 cpu‚25.210318498916088 network.recv‚600679 disk.rbd2.in‚0 disk.sda1.in‚0 disk.md0p1.in‚ 0.0703125 disk.sdb1.in‚0! disk.sdb2.out‚192.81103515625 disk.dm-0.in‚0.078125# disk.nvme0n1.out‚10470.08984375 disk.sdb.in‚0.03125% disk./.usageGB‚2711.4373054504395 disk.sda.in‚ 0.0390625& proc.memory.availableMB‚ 5.67578125) disk./.usagePercent‚75.81292188506875 disk.sda.out‚192.81103515625 disk.sda2.in‚ 0.0390625 disk.sdb.out‚192.81103515625* proc.memory.percent‚18.288731575012207)¢sÊ2°- _wandbruntime‚1697  _runtime‚1697oŸ¤§Ä±¾ % lr-AdamW/pg1‚0.000856671431754553 % lr-AdamW/pg2‚0.000856671431754553 trainer/global_step‚1449 $ _timestamp‚1.788512072313053e+09 _runtime‚1697.783521844 _step‚115ÂÚº Ȳ _runtime‚1697.783521844 _step‚115 % lr-AdamW/pg1‚0.000856671431754553 % lr-AdamW/pg2‚0.000856671431754553 trainer/global_step‚1449 $ _timestamp‚1.788512072313053e+09s]Õê€2³- _wandbruntime‚1697  _runtime‚1697|€h”ß ´æO VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚120509661266944 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚120508743857152 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007233 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚117866500350497 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚66123719 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚36298577 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8685192 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1638288120465 % _timestamp‚1.7885120723145196e+09 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚118864340768256 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚118864322765312 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚617 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26086 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚118864322765312 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31684 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚133026486069248 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚66149807 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31929 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚27231165 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚133026510009344 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1166 ZLDeviceStatsMonitor.on_train_batch_start/inactipœoD ve_split_bytes.large_pool.peak‚ 175228416 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1714436809216 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1191 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚29825144 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚120509661266944 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917409792 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917409792 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26086 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚134740698211840 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955590144 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899406848 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899406848 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177046016 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31684 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚36298577 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚36324663 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 398533632 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚117867414801269 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1644421091840 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450772 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚18545973 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1645320498688 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚27231784 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955590144 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 224666624 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177046016 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26088 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607ípbNð'µêO VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 248606720 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1645320498688 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚134740946818560 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚120508743857152 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚29825142 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚619 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1714212142592 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450772 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚18545973 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1645320498688 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚27231784 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955590144 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 224666624 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177046016 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26088 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚120509661266944 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚120508743857152 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007233 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚117866500350497 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚66123719 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚36298577 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8685192 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1638288120465 % _timestamp‚1.7885120723145196e+09 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚118864340768256 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚118864322765312 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚617 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26086 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚118864322765312 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31684 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚133026486069248 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚66149807 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31929 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚27231165 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚133026510009344 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1166 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1714436809216 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1191 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚29825144 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚120509661266944 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917409792 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917409792 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26086 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚134740698211840 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955590144 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899406848 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899406848 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177046016 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31684 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚36298577 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚36324663 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 398533632 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚117867414801269 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1644421091840 IDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955590144 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚29845040 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚117945589105943 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 _runtime‚1698.977616661 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚18558351 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚117946503556923 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411620 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1646433315328 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214657 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 trainer/global_step‚1449 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26114 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚29845040 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450980 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31929 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚66168208 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚36323168 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007233 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚29845042 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚118943953060864 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1645533895168 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1191 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 $ _timestamp‚1.788512073505531e+09 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚66168208 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917423104 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899420160 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚27249265 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚29845042 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚36323168 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚66194322 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚133115805289984 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1640294193403 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚36349280 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26112 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚120590404379136 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1646433315328 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚36349280 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚134831181385728 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177046016 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚120589486956032 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8691531 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728ô±M½¹¸óM \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1646433315328 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚36349280 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚134831181385728 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177046016 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚120589486956032 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8691531 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚118943953060864 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917423104 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚617 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899420160 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177046016 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955590144 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚29845040 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚117945589105943 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 _runtime‚1698.977616661 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚18558351 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚117946503556923 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411620 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1646433315328 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214657 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 trainer/global_step‚1449 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26114 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚29845040 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450980 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31929 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚66168208 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚36323168 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007233 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚29845042 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚118943953060864 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1645533895168 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1191 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 $ _timestamp‚1.788512073505531e+09 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚66168208 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917423104 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899420160 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚27249265 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚29845042 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚36323168 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚66194322 _KDeviceStatŒ‚$ @sMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚133115805289984 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1640294193403 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚36349280 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26112 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚120590404379136u”ì2¹- _wandbruntime‚1698  _runtime‚1698Ú]•WºÑ /loss/FRt4/assignment_loss‚4.149918854236603 1loss/SRqqt3/detection_loss‚1.6615165355852095 1loss/SRqqt1/assignment_loss‚2.227081023877667 2loss/SRqqt4/assignment_loss‚1.9828513418474505 0loss/FBt4/assignment_loss‚0.7935821626867566 /loss/FBt2/detection_loss‚0.6932227790909761 .loss/FBt3/detection_loss‚3.720820426940918 _step‚118 0loss/FRt2/assignment_loss‚1.8501971463362377 2loss/SRqqt2/assignment_loss‚1.0283420902466582 0loss/SRqqt1/detection_loss‚2.761828983022321 1loss/SRqqt4/detection_loss‚2.6394041180610657 .loss/FBt1/detection_loss‚2.063373419973585 _runtime‚1698.979966402 %loss/total_loss‚48.51630159910171 epoch‚1 0loss/FRt3/assignment_loss‚3.3858036158377662 0loss/FBt1/assignment_loss‚0.5701280267264539 /loss/FRt3/detection_loss‚2.2418621307925175 /loss/FRt4/detection_loss‚2.8401755736424374 1loss/SRqqt3/assignment_loss‚1.572215270408887 /loss/FRt1/detection_loss‚2.7138489464918774 /loss/FRt2/detection_loss‚1.6164881235195532 trainer/global_step‚1449 $ _timestamp‚1.788512073509615e+09 1loss/SRqqt2/detection_loss‚1.3392687409875854 /loss/FRt1/assignment_loss‚2.966926125685374 1loss/FBt2/assignment_loss‚0.27091158631880896 /loss/FBt3/assignment_loss‚0.590606485803922 /loss/FBt4/detection_loss‚2.8359280909810747ɉÒ[»Õ 1loss/SRqqt2/detection_loss‚1.3392687409875854 /loss/FRt1/assignment_loss‚2.966926125685374 1loss/FBt2/assignment_loss‚0.27091158631880896 /loss/FBt3/assignment_loss‚0.590606485803922 /loss/FBt4/detection_loss‚2.8359280909810747 /loss/FRt4/assignment_loss‚4.149918854236603 1loss/SRqqt3/detection_loss‚1.6615165355852095 1loss/SRqqt1/assignment_loss‚2.227081023877667 2loss/SRqqt4/assignment_loss‚1.9828513418474505 0loss/FBt4/assignment_loss‚0.7935821626867566 /loss/FBt2/detection_loss‚0.6932227790909761 .loss/FBt3/detection_loss‚3.720820426940918 _step‚118 0loss/FRt2/assignment_loss‚1.8501971463362377 2loss/SRqqt2/assignment_loss‚1.0283420902466582 0loss/SRqqt1/detection_loss‚2.761828983022321 1loss/SRqqt4/detection_loss‚2.6394041180610657 .loss/FBt1/detection_loss‚2.063373419973585 _runtime‚1698.979966402 %loss/total_loss‚48.51630159910171 epoch‚1 0loss/FRt3/assignment_loss‚3.3858036158377662 0loss/FBt1/assignment_loss‚0.5701280267264539 /loss/FRt3/detection_loss‚2.2418621307925175 /loss/FRt4/detection_loss‚2.8401755736424374 1loss/SRqqt3/assignment_loss‚1.572215270408887 /loss/FRt1/detection_loss‚2.7138489464918774 /loss/FRt2/detection_loss‚1.6164881235195532 trainer/global_step‚1449 $ _timestamp‚1.788512073509615e+09vóغR¼‚(:Ç ÔŽêÔñÔ disk.sdb.in‚0.03125 disk.sdb2.in‚0.03125" memory_percent‚99.965500831604& proc.memory.availableMB‚ 5.65234375 network.sent‚ 2.428976e+06$ disk./.usageGB‚2711.471019744873 disk.nvme1n1.out‚ 7742.328125 disk.md0.in‚ 0.0703125 disk.md0.out‚ 187.3671875 cpu‚25.20875342357335 disk.nvme0n1.in‚ 0.02734375 disk.sdb.out‚194.3974609375 disk.md0p1.in‚ 0.0703125* proc.memory.percent‚18.288731575012207 disk.sda.in‚ 0.0390625 disk.sda.out‚194.3974609375 disk.dm-0.out‚ 18374.6796875 disk.md0p1.out‚ 187.3671875 network.recv‚605190 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.sdb2.out‚194.3974609375 proc.cpu.threads‚17) disk./.usagePercent‚75.81386455085165" disk.nvme0n1.out‚ 10632.3515625 disk.sda1.out‚0 disk.dm-0.in‚0.078125# proc.memory.rssMB‚ 2996.42578125 disk.sda2.out‚194.3974609375 disk.rbd0.out‚ 0.279296875 disk.nvme1n1.in‚ 0.05078125 disk.rbd2.in‚0 disk.sda1.in‚0 disk.sda2.in‚ 0.0390625 disk.rbd0.in‚0 disk.rbd2.out‚ 0.50390625°k“½‚(:ˆ ÔŽêÔšô¼€ gpu.0.gpu‚25.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚37.0 gpu.0.powerWatts‚170.718' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.6795 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¿š’¾‚(:‡ ãŽêÔ“¬­ÿ gpu.0.gpu‚43.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚37.0 gpu.0.powerWatts‚128.388' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚32.097 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0RË“T¿‚(:É ãŽêÔÿ€€) disk./.usagePercent‚75.81648274693363 disk.nvme1n1.in‚ 0.05078125 disk.md0.in‚ 0.0703125 disk.sdb2.in‚0.03125 network.recv‚611218 disk.sdb1.in‚0 disk.sda1.in‚0 disk.md0p1.in‚ 0.0703125 disk.md0p1.out‚ 189.5078125 disk.sdb.in‚0.03125 disk.sdb.out‚196.60205078125 disk.sda.out‚196.60205078125 disk.sdb1.out‚0 disk.sda1.out‚0 disk.nvme0n1.in‚ 0.02734375! disk.sda2.out‚196.60205078125! proc.memory.availableMB‚5.375 cpu‚25.20728431155984 disk.dm-0.in‚0.078125" disk.nvme0n1.out‚ 10782.3359375 disk.sda.in‚ 0.0390625 disk.sda2.in‚ 0.0390625 disk.rbd0.in‚0# proc.memory.rssMB‚ 2996.42578125 proc.cpu.threads‚17 network.sent‚ 2.539027e+06 disk.nvme1n1.out‚ 7842.640625 disk.rbd2.in‚0 disk.dm-0.out‚ 18624.9765625 disk.rbd0.out‚ 0.279296875% disk./.usageGB‚2711.5646591186523$ memory_percent‚99.96719360351562 disk.md0.out‚ 189.5078125 disk.rbd2.out‚ 0.50390625! disk.sdb2.out‚196.60205078125* proc.memory.percent‚18.288731575012207)†G‰”À‚(:‰ òŽêÔ–ÕÂÿ gpu.0.gpu‚49.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚36.0 gpu.0.powerWatts‚114.131' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.53275 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0jð6G>Á‚(:³ òŽêÔÛ柀 disk.sdb1.in‚0 cpu‚25.20565731766835 disk.md0.out‚ 191.4296875 network.sent‚ 2.545278e+06 disk.sda.in‚ 0.0390625 disk.rbd0.in‚0 proc.cpu.threads‚17 disk.sdb.out‚ 198.609375 disk.nvme1n1.in‚ 0.05078125! disk.nvme1n1.out‚ 7969.5078125$ memory_percent‚99.96871948242188 disk.nvme0n1.in‚ 0.02734375 disk.sdb.in‚0.03125 disk.rbd0.out‚ 0.279296875! proc.memory.availableMB‚5.125* proc.memory.percent‚18.288731575012207 disk.md0.in‚ 0.0703125 disk.sdb2.in‚0.03125 disk.sdb2.out‚ 198.609375 disk.sda2.out‚ 198.609375 disk.sdb1.out‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-0.in‚0.078125 disk.md0p1.out‚ 191.4296875 disk.sda2.in‚ 0.0390625 disk.rbd2.in‚0) disk./.usagePercent‚75.81997683930541 disk.dm-0.out‚ 18895.96875 disk.sda.out‚ 198.609375# proc.memory.rssMB‚ 2996.42578125" disk.nvme0n1.out‚ 10926.4609375 disk.rbd2.out‚ 0.50390625 network.recv‚616157$ disk./.usageGB‚2711.689624786377 disk.md0p1.in‚ 0.07031258/7”‚(:‰ êÔçªÖÿ gpu.0.gpu‚35.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚36.0 gpu.0.powerWatts‚116.583' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚29.14575 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¦ÊßFÂ(:» êÔñú“€" disk.nvme1n1.out‚ 8005.09765625 disk.sdb1.out‚0 disk.md0.out‚ 192.890625 disk.rbd2.in‚0 disk.dm-0.out‚18987.52734375 disk.sda2.out‚ 200.140625 disk.rbd0.in‚0 disk.md0.in‚ 0.0703125 disk.sda1.out‚0 disk.md0p1.in‚ 0.0703125) disk./.usagePercent‚75.82696641063588 disk.rbd0.out‚ 0.279296875 disk.nvme1n1.in‚ 0.05078125 disk.sdb2.out‚ 200.140625 disk.md0p1.out‚ 192.890625 disk.sdb.out‚ 200.140625% proc.memory.availableMB‚ 5.3515625 network.sent‚ 2.548374e+06% disk./.usageGB‚2711.9396057128906 disk.rbd2.out‚ 0.50390625" disk.nvme0n1.out‚ 10982.4296875 disk.sdb.in‚0.03125 disk.sda.in‚ 0.0390625 disk.sdb2.in‚0.03125$ memory_percent‚99.96733665466309# proc.memory.rssMB‚ 2996.42578125 network.recv‚619765 disk.sdb1.in‚0 disk.sda1.in‚0 disk.dm-0.in‚0.078125 disk.nvme0n1.in‚ 0.02734375* proc.memory.percent‚18.288731575012207 cpu‚25.204208164417533 proc.cpu.threads‚17 disk.sda.out‚ 200.140625 disk.sda2.in‚ 0.0390625±Jix2Ä- _wandbruntime‚1756  _runtime‚1756äü_kÇÅÁ & lr-AdamW/pg1‚0.0008561982193289201 & lr-AdamW/pg2‚0.0008561982193289201 trainer/global_step‚1499 % _timestamp‚1.7885121311904836e+09 _runtime‚1756.660896976 _step‚119kïŽËÆÅ & lr-AdamW/pg1‚0.0008561982193289201 & lr-AdamW/pg2‚0.0008561982193289201 trainer/global_step‚1499 % _timestamp‚1.7885121311904836e+09 _runtime‚1756.660896976 _step‚119wÅ=ð22Ç- _wandbruntime‚1756  _runtime‚1756J_«ì'ÈæO XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899406848 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚121821866164777 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚68377609 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚28137275 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 244412416 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚37557565 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8971874 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1700075672064 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚122845855545856 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 220472320 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚124546830624768 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚122845855545856 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 trainer/global_step‚1499 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955590144 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚124545913214976 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31684 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚30820044 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1700975078912 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955590144 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 398533632 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26088 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 _step‚120 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚121820951714005 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214657 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1700075672064 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1191 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450772 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26088 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚68351521 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚124545913214976 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚122845837542912 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31929 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚30820042 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚120127233975792 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917409792 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚28136647 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚122845837542912 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31684 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚628 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚626 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚137492471046144 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 _runtime‚1756.661971811 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26086 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1772693478912 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚30820042 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚37531479 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚139265164525056 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899406848 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚30820044 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚120127216936432 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917409792 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1693734777573 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8972500 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚137492447106048 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1694632188985 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728h™|ò"ÉêO UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚122845837542912 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31684 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚628 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚626 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚137492471046144 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 _runtime‚1756.661971811 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26086 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1772693478912 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚30820042 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚37531479 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚139265164525056 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899406848 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚30820044 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚120127216936432 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917409792 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1693734777573 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8972500 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚137492447106048 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1694632188985 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899406848 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚121821866164777 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚68377609 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚28137275 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 244412416 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚37557565 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8971874 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1700075672064 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚122845855545856 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 220472320 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚124546830624768 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚122845855545856 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 trainer/global_step‚1499 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955590144 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚124545913214976 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31684 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚30820044 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1700975078912 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955590144 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 398533632 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26088 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 _step‚120 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚121820951714005 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214657 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1700075672064 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1191 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450772 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26088 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚68351521 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚124545913214976 \HDeviceStatsMonitor.on_tr°'Žëain_batch_start/allocated_bytes.large_pool.freed‚122845837542912 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31929 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚30820042 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚120127233975792 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917409792 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚28136647 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899420160 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚137581766326784 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411620 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 216264704 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚122925485841408 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 _runtime‚1757.594443379 _step‚121 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26114 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26112 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899420160 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚30839940 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚124627573736960 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917423104 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955590144 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1701188475392 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚28155430 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1773861307904 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚120205197030560 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚68396010 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1695740850511 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚68422124 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007233 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214657 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177046016 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚614 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚37556070 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1702087895552 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1166 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7885121321235707e+09 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚122925485841408 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚28154816 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚30839942 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917423104 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31684 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚121900954920431 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955590144 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚124626656313856 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚612 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26112 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 240204800 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 398533632 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171ÌNîéú&ÌôM [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 216264704 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚122925485841408 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 _runtime‚1757.594443379 _step‚121 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26114 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26112 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899420160 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚30839940 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚124627573736960 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917423104 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955590144 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1701188475392 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚28155430 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1773861307904 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚120205197030560 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚68396010 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1695740850511 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚68422124 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007233 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214657 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177046016 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚614 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚37556070 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1702087895552 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1166 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7885121321235707e+09 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚122925485841408 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚28154816 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚30839942 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917423104 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31684 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚121900954920431 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955590144 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚124626656313856 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚612 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26112 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 240204800 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 398533632 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1701188475392 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31929 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚30839942 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899420160 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚137581766326784 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411620yl¶D2Í- _wandbruntime‚1757  _runtime‚1757ÿø RWÎÑ _runtime‚1757.596283987 /loss/FRt3/assignment_loss‚3.365394535817598 2loss/SRqqt3/assignment_loss‚1.7955410021046798 0loss/FBt4/assignment_loss‚0.4951824563053938 /loss/FRt4/detection_loss‚2.9518685725427445 2loss/SRqqt1/assignment_loss‚1.4710824096067385 .loss/FBt1/assignment_loss‚0.28582385443151 1loss/SRqqt2/detection_loss‚1.0670158516680985 % _timestamp‚1.7885121321261625e+09 /loss/FRt1/assignment_loss‚3.512143886751599 1loss/SRqqt4/assignment_loss‚2.092705930857097 0loss/FBt2/assignment_loss‚0.2550160024398757 0loss/FRt2/assignment_loss‚1.7870279257757622 0loss/FRt4/assignment_loss‚3.8790443174300657 trainer/global_step‚1499 0loss/SRqqt1/detection_loss‚2.103394371812994 1loss/SRqqt4/detection_loss‚2.5005769492948757 .loss/FBt3/detection_loss‚3.335015797615051 .loss/FBt4/detection_loss‚2.178743440371293 .loss/FRt1/detection_loss‚2.351360418730312 /loss/FBt1/detection_loss‚2.2365474581718443 epoch‚1 1loss/FBt3/assignment_loss‚0.10600003278814256 /loss/FRt3/detection_loss‚2.1921187513752987 /loss/FBt2/detection_loss‚0.6236932842813908 &loss/total_loss‚45.342751788656955 _step‚122 2loss/SRqqt2/assignment_loss‚1.1635119200912876 /loss/FRt2/detection_loss‚1.7347078190038079 0loss/SRqqt3/detection_loss‚1.859234799389486u€¯[ÏÕ 1loss/SRqqt4/assignment_loss‚2.092705930857097 0loss/FBt2/assignment_loss‚0.2550160024398757 0loss/FRt2/assignment_loss‚1.7870279257757622 0loss/FRt4/assignment_loss‚3.8790443174300657 trainer/global_step‚1499 0loss/SRqqt1/detection_loss‚2.103394371812994 1loss/SRqqt4/detection_loss‚2.5005769492948757 .loss/FBt3/detection_loss‚3.335015797615051 .loss/FBt4/detection_loss‚2.178743440371293 .loss/FRt1/detection_loss‚2.351360418730312 /loss/FBt1/detection_loss‚2.2365474581718443 epoch‚1 1loss/FBt3/assignment_loss‚0.10600003278814256 /loss/FRt3/detection_loss‚2.1921187513752987 /loss/FBt2/detection_loss‚0.6236932842813908 &loss/total_loss‚45.342751788656955 _step‚122 2loss/SRqqt2/assignment_loss‚1.1635119200912876 /loss/FRt2/detection_loss‚1.7347078190038079 0loss/SRqqt3/detection_loss‚1.859234799389486 _runtime‚1757.596283987 /loss/FRt3/assignment_loss‚3.365394535817598 2loss/SRqqt3/assignment_loss‚1.7955410021046798 0loss/FBt4/assignment_loss‚0.4951824563053938 /loss/FRt4/detection_loss‚2.9518685725427445 2loss/SRqqt1/assignment_loss‚1.4710824096067385 .loss/FBt1/assignment_loss‚0.28582385443151 1loss/SRqqt2/detection_loss‚1.0670158516680985 % _timestamp‚1.7885121321261625e+09 /loss/FRt1/assignment_loss‚3.512143886751599zÉÝœŽ’Ђ(:‡ êÔ„ÀÈ€ gpu.0.gpu‚42.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚171.168' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.792 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ß–þOÑ‚(:Ä êÔþ–´% proc.memory.availableMB‚ 5.3203125 network.sent‚ 2.554366e+06% disk./.usageGB‚2711.3846473693848 disk.nvme0n1.in‚ 0.02734375 disk.sdb.out‚ 202.03515625 disk.sda.in‚ 0.0390625 disk.sda2.out‚ 202.03515625 disk.sda1.in‚0 disk.sda1.out‚0 disk.rbd0.in‚0 disk.dm-0.out‚ 19233.2109375$ memory_percent‚99.96752738952637 cpu‚25.202604523466604 network.recv‚624309 disk.sda.out‚ 202.03515625 disk.sdb1.out‚0 disk.nvme1n1.in‚ 0.05078125 disk.nvme1n1.out‚ 8178.546875 disk.sdb1.in‚0 disk.sdb2.out‚ 202.03515625) disk./.usagePercent‚75.81144954308334 disk.sdb.in‚0.03125 disk.rbd0.out‚ 0.279296875 disk.sdb2.in‚0.03125 disk.sda2.in‚ 0.0390625 disk.md0.in‚ 0.0703125 disk.md0.out‚ 194.71484375 disk.dm-0.in‚0.078125* proc.memory.percent‚18.288731575012207 proc.cpu.threads‚17 disk.md0p1.out‚ 194.71484375" disk.nvme0n1.out‚ 11054.6640625 disk.rbd2.in‚0# proc.memory.rssMB‚ 2996.42578125 disk.md0p1.in‚ 0.0703125 disk.rbd2.out‚ 0.50390625¬\l¶”Ò‚(:‰ ŸêÔ´èöÿ gpu.0.gpu‚17.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚39.0 gpu.0.powerWatts‚173.059' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.26475 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0!Ÿ`ýSÓ‚(:È ŸêԈꫀ disk.sda1.in‚0 disk.md0p1.out‚ 196.62890625 disk.nvme0n1.in‚ 0.02734375" disk.nvme0n1.out‚ 11138.7734375 disk.sdb.in‚0.03125 disk.sdb.out‚204.0107421875 network.recv‚629310 disk.dm-0.out‚ 19404.0234375 disk.md0.in‚ 0.0703125& proc.memory.availableMB‚ 5.32421875 proc.cpu.threads‚17 disk.sda1.out‚0 disk.sda.out‚204.0107421875 disk.nvme1n1.in‚ 0.05078125 disk.md0.out‚ 196.62890625 disk.sdb2.out‚204.0107421875# proc.memory.rssMB‚ 2996.42578125 disk.sda2.out‚204.0107421875 disk.rbd0.in‚0 disk.nvme1n1.out‚8265.25 disk.rbd2.in‚0$ memory_percent‚99.96750354766846 cpu‚25.201232915868694 disk.rbd0.out‚ 0.279296875$ disk./.usageGB‚2711.384323120117 disk.md0p1.in‚ 0.0703125 disk.sdb1.out‚0 disk.rbd2.out‚ 0.50390625 disk.dm-0.in‚0.078125 disk.sdb1.in‚0 network.sent‚ 2.663797e+06) disk./.usagePercent‚75.81144047693812 disk.sda.in‚ 0.0390625 disk.sda2.in‚ 0.0390625 disk.sdb2.in‚0.03125* proc.memory.percent‚18.288731575012207rMÉ“Ô‚(:ˆ ®êÔõ‡€ gpu.0.gpu‚51.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚174.286' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.5715 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0›,=^Õ‚(:Ó ®êÔՇŀ# proc.memory.rssMB‚ 2996.42578125 network.sent‚ 2.670091e+06 disk.md0.out‚ 198.14453125 disk.sda1.in‚0 disk.md0p1.out‚ 198.14453125 disk.rbd0.out‚ 0.279296875 disk.sda1.out‚0* proc.memory.percent‚18.288731575012207 disk.rbd2.out‚ 0.50390625! disk.sdb2.out‚205.60498046875 disk.dm-0.out‚19405.45703125 disk.sda.in‚ 0.0390625 disk.sda.out‚205.60498046875! disk.sda2.out‚205.60498046875" disk.nvme1n1.out‚ 8266.43359375$ memory_percent‚99.96895790100098% proc.memory.availableMB‚ 5.0859375 disk.sdb.out‚205.60498046875 disk.sdb1.in‚0 disk.md0.in‚ 0.0703125" disk.nvme0n1.out‚ 11139.0234375 disk.md0p1.in‚ 0.0703125 network.recv‚634145) disk./.usagePercent‚75.80904413476294% disk./.usageGB‚2711.2986183166504 disk.sdb1.out‚0 disk.rbd2.in‚0 disk.dm-0.in‚0.078125 disk.nvme0n1.in‚ 0.02734375 disk.sdb.in‚0.03125 disk.sda2.in‚ 0.0390625 disk.rbd0.in‚0 disk.nvme1n1.in‚ 0.05078125 disk.sdb2.in‚0.03125 cpu‚25.199710668608432 proc.cpu.threads‚17Ö„Åü2Ö- _wandbruntime‚1808  _runtime‚1808rüBuÇ×Á & lr-AdamW/pg1‚0.0008556359482374656 & lr-AdamW/pg2‚0.0008556359482374656 trainer/global_step‚1549 % _timestamp‚1.7885121827350047e+09 _runtime‚1808.205396195 _step‚123Üß äËØÅ _runtime‚1808.205396195 _step‚123 & lr-AdamW/pg1‚0.0008556359482374656 & lr-AdamW/pg2‚0.0008556359482374656 trainer/global_step‚1549 % _timestamp‚1.7885121827350047e+09{3(HG2Ù- _wandbruntime‚1808  _runtime‚1808~…¾Pâ ÚæO [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1756629659136 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26088 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚29043001 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9259426 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31684 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚19783575 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.sŽ+ºM mall_pool.peak‚1166 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚141958432082944 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31929 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26086 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚126827370323456 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚31814942 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1750078846093 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚128583082572800 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31929 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚29042413 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚70605411 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450772 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 trainer/global_step‚1549 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚126827352320512 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177046016 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚586 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚141958408142848 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚125776317528285 % _timestamp‚1.7885121827361076e+09 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚31814942 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚126827352320512 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31684 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚70579323 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9258840 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚125775403077513 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚128583082572800 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 398533632 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899406848 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚1808.206584911 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚19783573 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1749181434681 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26088 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚31814944 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955590144 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚38790467 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1830940487168 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1756629659136 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚143789372570112 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚128583999982592 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚128583999982592 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917409792 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1755730252288 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚38764381 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚588 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚38764381 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1830713723392 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899406848 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚31814944 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚124026221642832 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 250703872“5°¦ð'ÛêO F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚1808.206584911 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚19783573 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1749181434681 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26088 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚31814944 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955590144 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚38790467 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1830940487168 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1756629659136 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚143789372570112 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚128583999982592 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚128583999982592 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917409792 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1755730252288 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚38764381 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚588 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚38764381 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1830713723392 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899406848 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚31814944 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚124026221642832 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 250703872 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1756629659136 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26088 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚29043001 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9259426 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31684 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚19783575 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1166 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚141958432082944 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31929 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26086 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚126827370323456 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚31814942 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1750078846093 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚128583082572800 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31929 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚29042413 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚70605411 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450772 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 trainer/global_step‚1549 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚126827352320512 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177046016 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚586 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚141958408142848 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚125776317528285 % _timestamp‚1.7885121827361076e+09 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚31814942 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚126827352320512 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31684 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚70579323 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9258840 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚125775403077513 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚128583082572800 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 398533632 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899406848 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0|ºƒŠ2Ü- _wandbruntime‚1809  _runtime‚1809Ðç~ö&ÝðM E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚126907000619008 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚128663825671680 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955590144 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177046016 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214657 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9265216 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚125854491832959 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1751187507619 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26112 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚29060569 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚143879607952384 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1757742475776 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1831880588800 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007233 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚70649926 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚19795951 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 216264704 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31929 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚38815084 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚31834840 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 398533632 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚142047727363584 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31929 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚70649926 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚38788972 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚143879848157184 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 trainer/global_step‚1549 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955590144 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚128664743094784 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1756843055616 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚70623812 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚70623812 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26112 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚128664743094784 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚596 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1757742475776 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31684 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚38788972 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177046016 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚126906982616064 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899420160 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚29061167 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚19795949 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚31834840 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚126907000619008 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 240204800 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚142047751303680 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1832096853504 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚124104201736960 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917423104 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917423104 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899420160 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1756843055616 % _timestamp‚1.7885121836636581e+09 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1166 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚125855406283939 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171rØ—ý»ÞôM NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚31834842 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9264620 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚38815084 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214657 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9265216 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚125854491832959 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1751187507619 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26112 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚29060569 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚143879607952384 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1757742475776 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1831880588800 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007233 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚70649926 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚19795951 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 216264704 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31929 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚38815084 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚31834840 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 398533632 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚142047727363584 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31929 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚70649926 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚38788972 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚143879848157184 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 trainer/global_step‚1549 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955590144 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚128664743094784 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1756843055616 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚70623812 G:DÆ‚Qv?eviceStatsMonitor.on_train_batch_end/allocation.all.freed‚70623812 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26112 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚128664743094784 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚596 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1757742475776 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31684 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚38788972 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177046016 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚126906982616064 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899420160 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚29061167 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚19795949 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚31834840 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚126907000619008 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 240204800 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚142047751303680 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1832096853504 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚124104201736960 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917423104 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917423104 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899420160 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1756843055616 % _timestamp‚1.7885121836636581e+09 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1166 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚125855406283939 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚126907000619008 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚128663825671680 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955590144 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177046016 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 Fçì'îæO SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9546427 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚32809842 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚32809844 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚32809844 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31684 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚132620251930624 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚130808867098112 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 trainer/global_step‚1599 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31684 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚132620251930624 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚20402373 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1191 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚72807125 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚130808885101056 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411412 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26088 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚146424393119744 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177046016 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚127925243388592 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26086 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚39997283 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 224666624 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955590144 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1166 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 398533632 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955590144 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚130808867098112 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1812284239360 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31929 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26088 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚148313642514432 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899406848 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚72833213 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1812284239360 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1811384832512 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚40023369 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚623 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚20402375 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1805525503201 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450772 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1804628091789 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚621 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚127925226349232 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31684 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚132620251930624 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚130808867098112 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 trainer/global_step‚1599 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31684 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚132620251930624 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚20402373 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1191 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚72807125 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚130808885101056 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411412 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26088 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚146424393119744 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177046016 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚127925243388592 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26086 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚39997283 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 224666624 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955590144 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1166 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 398533632 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955590144 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚130808867098112 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1812284239360 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31929 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26088 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚148313642514432 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899406848 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚72833213 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1812284239360 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1811384832512 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚40023369 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚623 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚20402375 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1805525503201 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450772 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1804628091789 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚621 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚127925226349232 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚72877728 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31684 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214657 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚29967507 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚40047986 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚72851614 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚129809857647447 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1890194831360 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚128003206443360 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚72851614 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 _step‚129 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚40021874 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177046016 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚617 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1806634164727 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917423104 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚32829742 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚20414751 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚132700995029504 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899420160 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917423104 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31929 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚40047986 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚40021874 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚619 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚146513688400384 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚130888515396608 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1805736753107 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955590144 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚32829740 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚148403883231744 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚32829742 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚29966888 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26114 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚32829740 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚132701912452608 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1812497635840 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _runtime‚1862.210939297 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955590144 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚29©·û&òõM K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955590144 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1813397056000 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1812497635840 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚72877728 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31684 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214657 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚29967507 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚40047986 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚72851614 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚129809857647447 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1890194831360 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚128003206443360 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚72851614 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 _step‚129 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚40021874 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177046016 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚617 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1806634164727 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917423104 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚32829742 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚20414751 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚132700995029504 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899420160 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917423104 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31929 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚40047986 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚40021874 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚619 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚146513688400384 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚130888515396608 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1805736753107 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955590144 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚32829740 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚148403883231744 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚32829742 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚29966888 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26114 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚32829740 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚132701912452608 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1812497635840 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _runtime‚1862.210939297‘HÇÞ2ó- _wandbruntime‚1862  _runtime‚1862UÚV—VôÐ epoch‚1 trainer/global_step‚1599 1loss/FBt2/assignment_loss‚0.11117619157416334 0loss/SRqqt1/detection_loss‚1.925264931321144 0loss/FRt2/assignment_loss‚1.3686628856901395 .loss/FBt2/detection_loss‚0.855244314134423 $loss/total_loss‚46.3014852505939 $ _timestamp‚1.788512236742238e+09 _runtime‚1862.212371054 _step‚130 1loss/SRqqt2/detection_loss‚0.9470641414415749 /loss/FBt3/detection_loss‚2.6100118905305862 2loss/SRqqt4/assignment_loss‚2.6360560400145396 1loss/FBt4/assignment_loss‚0.26431015230627963 1loss/SRqqt1/assignment_loss‚1.813058398310095 2loss/SRqqt2/assignment_loss‚1.1462762281435597 /loss/FRt2/detection_loss‚1.6767775727025533 2loss/SRqqt3/assignment_loss‚1.8770385950182875 .loss/FBt1/detection_loss‚2.111351175741716 0loss/FBt1/assignment_loss‚0.1875596219165707 1loss/SRqqt4/detection_loss‚2.6291642763784955 .loss/FBt4/detection_loss‚4.051300500830014 .loss/FRt1/detection_loss‚2.239496782726171 0loss/FRt1/assignment_loss‚3.3007945892762165 0loss/FRt3/assignment_loss‚3.2762746956290267 /loss/FRt4/assignment_loss‚3.612314708305128 1loss/FBt3/assignment_loss‚0.44177148898597807 -loss/FRt4/detection_loss‚3.05452108744419 .loss/FRt3/detection_loss‚2.493494391441345 1loss/SRqqt3/detection_loss‚1.6725005907317003v·>,[õÕ .loss/FRt3/detection_loss‚2.493494391441345 1loss/SRqqt3/detection_loss‚1.6725005907317003 epoch‚1 trainer/global_step‚1599 1loss/FBt2/assignment_loss‚0.11117619157416334 0loss/SRqqt1/detection_loss‚1.925264931321144 0loss/FRt2/assignment_loss‚1.3686628856901395 .loss/FBt2/detection_loss‚0.855244314134423 $loss/total_loss‚46.3014852505939 $ _timestamp‚1.788512236742238e+09 _runtime‚1862.212371054 _step‚130 1loss/SRqqt2/detection_loss‚0.9470641414415749 /loss/FBt3/detection_loss‚2.6100118905305862 2loss/SRqqt4/assignment_loss‚2.6360560400145396 1loss/FBt4/assignment_loss‚0.26431015230627963 1loss/SRqqt1/assignment_loss‚1.813058398310095 2loss/SRqqt2/assignment_loss‚1.1462762281435597 /loss/FRt2/detection_loss‚1.6767775727025533 2loss/SRqqt3/assignment_loss‚1.8770385950182875 .loss/FBt1/detection_loss‚2.111351175741716 0loss/FBt1/assignment_loss‚0.1875596219165707 1loss/SRqqt4/detection_loss‚2.6291642763784955 .loss/FBt4/detection_loss‚4.051300500830014 .loss/FRt1/detection_loss‚2.239496782726171 0loss/FRt1/assignment_loss‚3.3007945892762165 0loss/FRt3/assignment_loss‚3.2762746956290267 /loss/FRt4/assignment_loss‚3.612314708305128 1loss/FBt3/assignment_loss‚0.44177148898597807 -loss/FRt4/detection_loss‚3.05452108744419‚ï‘vú”ö‚(:‰ ùêÔççÛþ gpu.0.gpu‚54.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚164.039' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚41.00975 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0þôóV÷‚(:Ë ùêÔ°âéþ cpu‚25.19279744593057 network.sent‚ 2.798055e+06 disk.sdb1.out‚0 disk.rbd2.out‚ 0.50390625 disk.sda.in‚ 0.0390625 disk.sda.out‚214.1689453125 network.recv‚655691 disk.sda2.in‚ 0.0390625 disk.sdb2.out‚214.1689453125 disk.md0p1.out‚ 206.41015625 disk.sdb.out‚214.1689453125 proc.cpu.threads‚17" disk.nvme1n1.out‚ 9092.44140625 disk.rbd2.in‚0$ memory_percent‚99.96743202209473 disk.rbd0.out‚ 0.279296875 disk.sdb1.in‚0 disk.dm-0.in‚0.078125 disk.dm-0.out‚20231.83203125 disk.md0p1.in‚ 0.0703125! disk.nvme0n1.out‚ 11139.390625% proc.memory.availableMB‚ 5.3359375) disk./.usagePercent‚75.81999017187194$ disk./.usageGB‚2711.690101623535 disk.sdb2.in‚0.03125 disk.sda1.in‚0 disk.nvme0n1.in‚ 0.02734375 disk.md0.out‚ 206.41015625 disk.sda1.out‚0 disk.sda2.out‚214.1689453125 disk.rbd0.in‚0 disk.nvme1n1.in‚ 0.05078125 disk.md0.in‚ 0.0703125 disk.sdb.in‚0.03125" proc.memory.rssMB‚ 2996.4296875* proc.memory.percent‚18.288755416870117ù¬–K“ø‚(:ˆ ˆêÔÉú€ gpu.0.gpu‚4.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚36.0 gpu.0.powerWatts‚114.131' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.53275 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ëmÏOù‚(:Ä ˆêÔŽÅ’ƒ disk.md0p1.out‚ 207.96484375 disk.sda.out‚ 215.80078125 disk.rbd0.out‚ 0.279296875 disk.md0.in‚ 0.0703125 disk.sdb1.out‚0 disk.sdb2.out‚ 215.80078125 disk.sda1.in‚0 disk.dm-0.in‚0.078125 disk.sdb.out‚ 215.80078125 disk.rbd0.in‚0% proc.memory.availableMB‚ 5.0859375 disk.md0.out‚ 207.96484375 disk.sdb.in‚0.03125 disk.nvme1n1.out‚ 9277.234375* proc.memory.percent‚18.288755416870117 disk.sda1.out‚0 disk.rbd2.out‚ 0.50390625$ memory_percent‚99.96895790100098 network.recv‚660972 disk.dm-0.out‚20416.67578125 disk.sda.in‚ 0.0390625 disk.nvme1n1.in‚ 0.05078125 disk.rbd2.in‚0) disk./.usagePercent‚75.81999017187194 disk.sdb2.in‚0.03125 disk.sda2.out‚ 215.80078125 proc.cpu.threads‚17 network.sent‚ 2.907637e+06 disk.md0p1.in‚ 0.0703125 disk.nvme0n1.in‚ 0.02734375# disk.nvme0n1.out‚11139.44140625 disk.sda2.in‚ 0.0390625 disk.sdb1.in‚0" proc.memory.rssMB‚ 2996.4296875$ disk./.usageGB‚2711.690101623535 cpu‚25.191561764143735 ã'Ë“ú‚(:ˆ —êԨɼÿ gpu.0.gpu‚2.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚168.383' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.09575 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚06A:AKû‚(:À —êÔ–ùðÿ disk.md0.in‚ 0.0703125 disk.rbd2.out‚ 0.50390625 disk.sdb2.in‚0.03125$ memory_percent‚99.96907711029053* proc.memory.percent‚18.288755416870117 disk.sda2.in‚ 0.0390625 disk.sda2.out‚ 217.27734375 disk.md0.out‚ 209.37890625 disk.rbd2.in‚0 disk.sdb2.out‚ 217.27734375 disk.dm-0.out‚ 20592.8125 disk.sda.out‚ 217.27734375 disk.sdb1.out‚0 cpu‚25.19016126195652 network.sent‚ 2.913451e+06$ disk./.usageGB‚2711.940158843994 disk.sdb.out‚ 217.27734375& proc.memory.availableMB‚ 5.06640625 network.recv‚665201 disk.rbd0.out‚ 0.279296875 disk.sdb1.in‚0) disk./.usagePercent‚75.82698187641303 disk.sda1.in‚0 disk.dm-0.in‚0.078125 disk.sdb.in‚0.03125! disk.nvme1n1.out‚ 9453.3671875 disk.md0p1.in‚ 0.0703125" proc.memory.rssMB‚ 2996.4296875 proc.cpu.threads‚17" disk.nvme0n1.out‚ 11139.4453125 disk.rbd0.in‚0 disk.nvme1n1.in‚ 0.05078125 disk.md0p1.out‚ 209.37890625 disk.sda1.out‚0 disk.nvme0n1.in‚ 0.02734375 disk.sda.in‚ 0.0390625,É&*2ü- _wandbruntime‚1915  _runtime‚1915J=µÇýÁ & lr-AdamW/pg1‚0.0008542447172981757 & lr-AdamW/pg2‚0.0008542447172981757 trainer/global_step‚1649 % _timestamp‚1.7885122904345908e+09 _runtime‚1915.904954535 _step‚131ë"*•ÌþÆ & lr-AdamW/pg1‚0.0008542447172981757 & lr-AdamW/pg2‚0.0008542447172981757 trainer/global_step‚1649 % _timestamp‚1.7885122904345908e+09 _runtime‚1915.904954535 _step‚131ƒX¾¿`2ÿ- _wandbruntime‚1915  _runtime‚1915¶€ÿÍ €æO G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚134790381875712 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚75061015 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31684 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955590144 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 244412416 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1867938819584 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚33804742 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚134790381875712 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31929 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9835658 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚133685220255301 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177046016 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1867039412736 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1174 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚136657421288448 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899406848 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚30856214 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 trainer/global_step‚1649 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26088 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 XKDeviceStatsMonitor.on_train_batch_start/ ðÙinactive_split.large_pool.allocated‚21021175 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚33804744 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 220472320 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31929 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1867938819584 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9835041 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚1915.906260891 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚152837832254464 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚75034927 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚41256271 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899406848 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177046016 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚41256271 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31684 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 398533632 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26088 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 _step‚132 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚41230185 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚30856833 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚134790399878656 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚150890330216448 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214657 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917409792 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚152837587842048 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411412 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1867039412736 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917409792 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚133684305804529 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 % _timestamp‚1.7885122904358065e+09 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26086 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚136657421288448 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955590144 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚131824231055632 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1947478097920 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1860074748897—ñ'ëO F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚1915.906260891 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚152837832254464 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚75034927 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚41256271 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899406848 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177046016 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚41256271 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31684 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 398533632 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26088 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 _step‚132 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚41230185 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚30856833 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚134790399878656 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚150890330216448 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214657 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917409792 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚152837587842048 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411412 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1867039412736 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917409792 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚133684305804529 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 % _timestamp‚1.7885122904358065e+09 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26086 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚136657421288448 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955590144 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚131824231055632 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1947478097920 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1860074748897 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚134790381875712 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚75061015 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31684 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955590144 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 244412416 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1867938819584 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚33804742 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚134790381875712 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31929 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9835658 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚133685220255301 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177046016 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1867039412736 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1174 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚136657421288448 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899406848 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚30856214 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 trainer/global_step‚1649 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26088 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚21021175 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚33804744 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 220472320 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31929 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1867938819584 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9835041 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416„‡£?h2‚- _wandbruntime‚1916  _runtime‚1916SÂ’_ö&ƒðM TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007233 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955590144 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7885122915195112e+09 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚41254776 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899420160 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1948657913856 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1861183410215 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31684 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1948456329216 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚133764309010955 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450980 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚75079416 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917423104 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚134870012171264 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚75079416 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1868152216064 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899420160 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚33824642 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚33824640 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411620 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955590144 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚603 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚152928105766400 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚150979649437184 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚33824642 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214657 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚134870030174208 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚75105530 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚136739081810432 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚30875068 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚150979673377280 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚136739081810432 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚41280888 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚605 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚33824640 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚134870030174208 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177046016 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 trainer/global_step‚1649 _runtime‚1916.991734814 _step‚133 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26114 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177046016 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚136738164387328 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚41254776 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚30874463 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚134870012171264 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1869051636224 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 225524736 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26112 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31929 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1199 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚41280888 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚75105530 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26112 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚152928331291136 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9840914 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚21033549 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9841517 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26112 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31929 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1199 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚41280888 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚75105530 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26112 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚152928331291136 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9840914 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚21033549 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9841517 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955590144 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7885122915195112e+09 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚41254776 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899420160 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1948657913856 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1861183410215 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31684 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1948456329216 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚133764309010955 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450980 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚75079416 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917423104 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚134870012171264 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚75079416 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1868152216064 VHDeviceStatsMonito«ÌL"Vr.on_train_batch_end/allocated_bytes.small_pool.current‚ 899420160 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚33824642 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚33824640 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411620 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955590144 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚603 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚152928105766400 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚150979649437184 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚33824642 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214657 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚134870030174208 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚75105530 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚136739081810432 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚30875068 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚150979673377280 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚136739081810432 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚41280888 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚605 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚33824640 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚134870030174208 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177046016 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 trainer/global_step‚1649 _runtime‚1916.991734814 _step‚133 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26114 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177046016 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚136738164387328 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚41254776 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚30874463…|bUr2…- _wandbruntime‚1916  _runtime‚1916uâ©«X†Ò 0loss/SRqqt4/detection_loss‚2.213191482950659 $loss/total_loss‚45.6302245017353 _runtime‚1916.994201696 2loss/FBt3/assignment_loss‚0.028140180744230747 epoch‚1 0loss/FRt3/assignment_loss‚2.8793859437659934 2loss/SRqqt4/assignment_loss‚1.9082025709178518 /loss/FRt3/detection_loss‚2.2387801243199243 _step‚134 0loss/FRt4/assignment_loss‚3.7184777089527676 1loss/SRqqt1/detection_loss‚2.2408571001142263 % _timestamp‚1.7885122915237746e+09 .loss/FRt4/detection_loss‚2.898165272814887 0loss/SRqqt3/detection_loss‚2.117424563485749 /loss/FBt2/detection_loss‚0.6704159443970861 /loss/FRt1/assignment_loss‚3.136025849580765 1loss/SRqqt2/detection_loss‚0.9744160851274711 -loss/FBt3/detection_loss‚2.32404034336408 trainer/global_step‚1649 0loss/FRt2/assignment_loss‚1.4652069466454642 1loss/SRqqt1/assignment_loss‚1.634092018629114 1loss/FBt2/assignment_loss‚0.07619771102798896 0loss/FBt4/assignment_loss‚0.4396770896273665 /loss/FRt1/detection_loss‚2.2838663411140443 .loss/FBt4/detection_loss‚5.060599505901337 2loss/SRqqt2/assignment_loss‚0.9186067945067433 2loss/SRqqt3/assignment_loss‚1.8476452299070603 /loss/FBt1/assignment_loss‚0.351111872328147 /loss/FBt1/detection_loss‚2.7668057547675238 /loss/FRt2/detection_loss‚1.4388920667448215>*(P]‡× /loss/FRt1/assignment_loss‚3.136025849580765 1loss/SRqqt2/detection_loss‚0.9744160851274711 -loss/FBt3/detection_loss‚2.32404034336408 trainer/global_step‚1649 0loss/FRt2/assignment_loss‚1.4652069466454642 1loss/SRqqt1/assignment_loss‚1.634092018629114 1loss/FBt2/assignment_loss‚0.07619771102798896 0loss/FBt4/assignment_loss‚0.4396770896273665 /loss/FRt1/detection_loss‚2.2838663411140443 .loss/FBt4/detection_loss‚5.060599505901337 2loss/SRqqt2/assignment_loss‚0.9186067945067433 2loss/SRqqt3/assignment_loss‚1.8476452299070603 /loss/FBt1/assignment_loss‚0.351111872328147 /loss/FBt1/detection_loss‚2.7668057547675238 /loss/FRt2/detection_loss‚1.4388920667448215 0loss/SRqqt4/detection_loss‚2.213191482950659 $loss/total_loss‚45.6302245017353 _runtime‚1916.994201696 2loss/FBt3/assignment_loss‚0.028140180744230747 epoch‚1 0loss/FRt3/assignment_loss‚2.8793859437659934 2loss/SRqqt4/assignment_loss‚1.9082025709178518 /loss/FRt3/detection_loss‚2.2387801243199243 _step‚134 0loss/FRt4/assignment_loss‚3.7184777089527676 1loss/SRqqt1/detection_loss‚2.2408571001142263 % _timestamp‚1.7885122915237746e+09 .loss/FRt4/detection_loss‚2.898165272814887 0loss/SRqqt3/detection_loss‚2.117424563485749 /loss/FBt2/detection_loss‚0.6704159443970861†(¬5­”ˆ‚(:‰ ¦êÔÓËãý gpu.0.gpu‚51.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚164.039' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚41.00975 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0F%"S‰‚(:È ¦êÔÔšòþ disk.rbd0.out‚ 0.279296875" disk.nvme1n1.out‚ 9750.76171875$ memory_percent‚99.97386932373047# proc.memory.availableMB‚4.28125$ disk./.usageGB‚2711.940101623535 disk.rbd2.in‚0! disk.sdb2.out‚219.24169921875 disk.sdb.in‚0.03125 disk.nvme1n1.in‚ 0.05078125" proc.memory.rssMB‚ 2996.4296875 disk.sda1.in‚0 disk.dm-0.in‚0.078125 disk.sda2.in‚ 0.0390625 disk.nvme0n1.in‚ 0.02734375* proc.memory.percent‚18.288755416870117 disk.rbd2.out‚ 0.50390625 disk.sda1.out‚0 disk.md0.in‚ 0.0703125 disk.nvme0n1.out‚11139.5! disk.sda2.out‚219.24169921875 cpu‚25.18897850744055 disk.md0.out‚ 211.26953125 disk.sda.out‚219.24169921875 proc.cpu.threads‚17 disk.rbd0.in‚0 network.recv‚669727) disk./.usagePercent‚75.82698027650505 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.md0p1.out‚ 211.26953125 disk.sdb.out‚219.24169921875 disk.sda.in‚ 0.0390625 network.sent‚ 2.917526e+06 disk.sdb2.in‚0.03125 disk.dm-0.out‚20890.26171875 disk.md0p1.in‚ 0.0703125)Å”Š‚(:‰ µêÔ«þ gpu.0.gpu‚56.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚36.0 gpu.0.powerWatts‚114.131' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.53275 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0!žYrU‹‚(:Ê µêÔ™ÒÏÿ disk.md0.in‚ 0.0703125 disk.dm-0.in‚0.078125 proc.memory.availableMB‚4.75 network.recv‚674294 disk.sdb.in‚0.03125 disk.rbd0.out‚ 0.279296875 disk.rbd2.out‚ 0.50390625 disk.sda1.in‚0 disk.md0p1.in‚ 0.0703125! disk.nvme0n1.out‚ 11139.703125% disk./.usageGB‚2711.8648414611816! disk.sda2.out‚220.98291015625 disk.sda1.out‚0" proc.memory.rssMB‚ 2996.4296875 disk.sda.out‚220.98291015625 network.sent‚ 3.024176e+06 disk.sdb2.in‚0.03125 disk.md0p1.out‚ 212.9453125 disk.nvme0n1.in‚ 0.02734375* proc.memory.percent‚18.288755416870117 disk.nvme1n1.in‚ 0.05078125 disk.dm-0.out‚21126.09765625$ memory_percent‚99.97100830078125 cpu‚25.187768286637713 disk.sda.in‚ 0.0390625 disk.sda2.in‚ 0.0390625 disk.rbd0.in‚0 disk.rbd2.in‚0! disk.sdb2.out‚220.98291015625) disk./.usagePercent‚75.82487597086683" disk.nvme1n1.out‚ 9986.39453125 disk.sdb1.in‚0 disk.md0.out‚ 212.9453125 proc.cpu.threads‚17 disk.sdb.out‚220.98291015625 disk.sdb1.out‚0©2“Œ‚(:ˆ ÄêÔ’©Úÿ gpu.0.gpu‚46.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚37.0 gpu.0.powerWatts‚149.442' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚37.3605 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¸ßx*V‚(:Ë ÄêÔš®Ô' disk./.usagePercent‚75.812225391794# disk.nvme1n1.out‚10115.37109375$ memory_percent‚99.96950626373291" proc.memory.rssMB‚ 2996.4296875 disk.sdb.out‚222.5029296875 disk.sda.in‚ 0.0390625 disk.sdb2.in‚0.03125& proc.memory.availableMB‚ 4.99609375* proc.memory.percent‚18.288755416870117 disk.sdb.in‚0.03125 disk.sda.out‚222.5029296875 disk.rbd0.out‚ 0.279296875 disk.nvme1n1.in‚ 0.05078125 disk.sdb2.out‚222.5029296875 disk.nvme0n1.in‚ 0.02734375 cpu‚25.18654289381372 disk.sda2.in‚ 0.0390625 disk.rbd2.out‚ 0.50390625 disk.dm-0.out‚ 21255.234375# disk.nvme0n1.out‚11139.86328125 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.md0.in‚ 0.0703125 disk.md0.out‚ 214.40234375 disk.rbd2.in‚0 disk.dm-0.in‚0.078125 proc.cpu.threads‚17$ disk./.usageGB‚2711.412395477295 disk.md0p1.out‚ 214.40234375 network.sent‚ 3.032093e+06 disk.rbd0.in‚0 disk.md0p1.in‚ 0.0703125 network.recv‚678366 disk.sda2.out‚222.5029296875 disk.sda1.in‚0 disk.sda1.out‚0Œ‰e½’Ž‚(:‡ ÓêÔÞ–Öÿ gpu.0.gpu‚30.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚163.09' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚40.7725 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0óyÔƒZ‚(:Ï ÓêÔŽ©“€ disk.sdb2.in‚0.03125 disk.sdb.in‚0.03125 disk.sdb1.in‚0 disk.rbd0.in‚0 network.recv‚682431) disk./.usagePercent‚75.81080478016668 disk.dm-0.in‚0.078125 disk.sda.out‚224.06005859375* proc.memory.percent‚18.288755416870117! disk.sdb2.out‚224.06005859375 disk.sda1.in‚0 disk.sda.in‚ 0.0390625 disk.nvme1n1.in‚ 0.05078125 disk.sdb1.out‚0$ memory_percent‚99.96874332427979 disk.md0p1.in‚ 0.0703125 disk.md0.in‚ 0.0703125" proc.memory.rssMB‚ 2996.4296875 cpu‚25.18517504529171 proc.cpu.threads‚17 disk.sda1.out‚0" disk.nvme0n1.out‚ 11139.8671875 disk.sdb.out‚224.06005859375 disk.sda2.in‚ 0.0390625! disk.sda2.out‚224.06005859375 disk.md0.out‚ 215.90234375 disk.rbd2.out‚ 0.50390625 network.sent‚ 3.035157e+06 disk.dm-0.out‚ 21256.046875 disk.md0p1.out‚ 215.90234375 disk.nvme0n1.in‚ 0.02734375 disk.rbd2.in‚0& proc.memory.availableMB‚ 5.12109375$ disk./.usageGB‚2711.361587524414 disk.rbd0.out‚ 0.279296875" disk.nvme1n1.out‚ 10116.1796875M¡ÆÎ2- _wandbruntime‚1973  _runtime‚1973 ·ÓxÆ‘À _runtime‚1973.785663946 _step‚135 & lr-AdamW/pg1‚0.0008534160473807827 & lr-AdamW/pg2‚0.0008534160473807827 trainer/global_step‚1699 $ _timestamp‚1.788512348315226e+09À]`öË’Å $ _timestamp‚1.788512348315226e+09 _runtime‚1973.785663946 _step‚135 & lr-AdamW/pg1‚0.0008534160473807827 & lr-AdamW/pg2‚0.0008534160473807827 trainer/global_step‚1699‡9Ö_„2“- _wandbruntime‚1973  _runtime‚1973`ðÞí'”çO YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚140695508056064 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31929 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚31765239 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1916418817417 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚77262729 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955590144 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚34799644 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚77288817 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214657 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚42463087 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31684 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚140695508056064 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177046016 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚138771896653312 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚140694590646272 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1923593399808 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚140694590646272 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1923593399808 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26088 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26088 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1174 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917409792 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚135723235762032 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚34799642 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007233 % _timestamp‚1.7885123483159993e+09 _runtime‚ 1973.78651862 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚34799644 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26086 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚157362052920320 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚155356291253248 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 trainer/global_step‚1699 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚77288817 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214657 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚42463087 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31684 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚140695508056064 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177046016 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚138771896653312 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚140694590646272 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1923593399808 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚140694590646272 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1923593399808 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26088 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26088 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1174 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917409792 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚135723235762032 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚34799642 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007233 % _timestamp‚1.7885123483159993e+09 _runtime‚ 1973.78651862 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚34799644 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26086 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚157362052920320 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚155356291253248 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 trainer/global_step‚1699 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 IDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955590144 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚42513790 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 248593408 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 224653312 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214657 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917423104 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899420160 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 _runtime‚1974.863610686 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚31782725 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚138851526948864 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚138851544951808 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚138851526948864 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚31783349 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚622 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1916630067323 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31929 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 _step‚137 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚77307218 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411620 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26112 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450980 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚34819540 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1923806796288 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚34819540 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚135801215856160 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚137717845923483 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚21652349 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1923806796288 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2006894932992 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 trainer/global_step‚1699 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚42487678 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31684 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1199 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177046016 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚140776251168256 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2006670279680 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚135801232895520 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917423104 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚77333332 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26114 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955590144 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177046016 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31929 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26112 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚138851544951808 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1924706216448 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚21652351 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚77307218 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 % _timestamp‚1.7885123493912995e+09 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚42513790 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚42487678 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1917527478943 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚77333332 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚157452529347072 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚34819542 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955590144 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚42513790 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 248593408 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 224653312 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214657 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917423104 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899420160 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 _runtime‚1974.863610686 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚31782725 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚138851526948864 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚138851544951808 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚138851526948864 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚31783349 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚622 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1916630067323 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31929 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 _step‚137 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚77307218 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411620 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26112 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450980 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚34819540 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1923806796288 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚34819540 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚135801215856160 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚137717845923483 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚21652349 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1923806796288 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2006894932992 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 trainer/global_step‚1699 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚42487678 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31684 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1199 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177046016 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚140776251168256 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2006670279680 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚135801232895520 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917423104 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚77333332 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26114 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955590144 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177046016 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31929 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26112 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚138851544951808 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1924706216448 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚21652351 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚77307218 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 % _timestamp‚1.7885123493912995e+09 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚42513790 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚42487678 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1917527478943 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚77333332 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚157452529347072 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚34819542 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26088 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10414249 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899406848 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177046016 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚144732677413888 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917409792 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007233 _runtime‚2027.508133211 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚22258773 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10413612 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1971865474525 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚144732677413888 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚22258775 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚159822276230144 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1978348573184 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 trainer/global_step‚1749 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1174 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚35794542 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955590144 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚139622240468432 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31684 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1979247980032 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1979247980032 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31929 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411412 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 252801024 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚139622257507792 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚35794544 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2063728601088 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚35794542 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚43695989 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚144731760004096 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚43695989 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚142753429433856 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚43722075 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚32673024 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚637 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚161886233692160 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2063957462016“tS=ò'©ìO LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917409792 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚144731760004096 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 398533632 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚142753429433856 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚142753411430912 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚32672385 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26088 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10414249 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899406848 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177046016 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚144732677413888 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917409792 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007233 _runtime‚2027.508133211 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚22258773 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10413612 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1971865474525 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚144732677413888 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚22258775 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚159822276230144 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1978348573184 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 trainer/global_step‚1749 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1174 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚35794542 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955590144 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚139622240468432 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31684 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1979247980032 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1979247980032 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31929 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411412 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 252801024 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚139622257507792 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚35794544 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2063728601088 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚35794542 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚43695989 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚144731760004096 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚43695989 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚142753429433856 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚43722075 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚32673024 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚637 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚161886233692160 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2063957462016 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1970968063113 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚141594122982317 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984Œþ(OH2ª- _wandbruntime‚2028  _runtime‚2028GrKí÷&«ñM G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚79535020 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31929 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚79561134 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1979461376512 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177046016 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917423104 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚142833059729408 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚161976699579904 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _runtime‚ 2028.44743946 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚35814442 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚144812503102976 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10419920 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚79535020 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955590144 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214657 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚144813420526080 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚142833041726464 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚35814442 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚22271151 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚35814440 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚43746692 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚622 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 398533632 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2065104129024 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚144813420526080 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2064875281408 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚43720580 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚620 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚159911595450880 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1972974136051 _step‚141 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚32691071 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚139700220562560 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚139700237601920 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚142833041726464 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚79561134 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411620 % _timestamp‚1.7885124029764965e+09 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚43746692 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26112 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚35814440 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31684 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899420160 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899420160 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450980 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1979461376512 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚32690449 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚161976446792192 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚141672297286991 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728lJˆ½Œ¬öM F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1199 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚22271149 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 228847616 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26112 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955590144 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214657 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚144813420526080 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚142833041726464 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚35814442 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚22271151 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚35814440 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚43746692 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚622 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 398533632 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2065104129024 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚144813420526080 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2064875281408 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚43720580 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚620 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚159911595450880 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1972974136051 _step‚141 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚32691071 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚139700220562560 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚139700237601920 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚142833041726464 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚79561134 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411620 % _timestamp‚1.78851*‚J8p24029764965e+09 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚43746692 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26112 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚35814440 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31684 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899420160 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899420160 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450980 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1979461376512 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚32690449 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚161976446792192 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚141672297286991 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚79535020 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31929 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚79561134 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1979461376512 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177046016 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917423104 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚142833059729408 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚161976699579904 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _runtime‚ 2028.44743946 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚35814442 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚144812503102976 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10419920 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚79535020 EDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177046016 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899406848 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚33580101 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚44928891 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚166410345081856 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2122348033024 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899406848 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 240218112 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚33580712 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 398533632 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2027312131633 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 trainer/global_step‚1799 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 216278016 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917409792 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚146734944211456 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚148768929361920 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚36789442 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917409792 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31684 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26088 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955590144 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚146734926208512 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚148769846771712 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚611 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚81718333 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26086 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177046016 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚609 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚164288237266944 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚44928891 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚145547659895053 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2034902560256 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 _step‚144 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚36789442 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚148769846771712 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚164288213326848 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 % _timestamp‚1.7885124546802757e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31929 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚148768929361920 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚145548574345825 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450772 _runtime‚2080.150920825 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚22877573 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 æÌXŠ!»íO L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚36789442 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚148769846771712 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚164288213326848 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 % _timestamp‚1.7885124546802757e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31929 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚148768929361920 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚145548574345825 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450772 _runtime‚2080.150920825 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚22877573 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2034902560256 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1204 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1179 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007233 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚36789444 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚146734944211456 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31684 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2034003153408 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚81744421 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2026414720221 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚36789444 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10702528 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177046016 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899406848 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚33580101 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚44928891 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚166410345081856 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2122348033024 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899406848 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 240218112 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚33580712 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 398533632 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2027312131633 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 trainer/global_step‚1799 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 216278016 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917409792 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚146734944211456 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚148768929361920 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚36789442 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917409792 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31684 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26088 JDDeviceStatsMonitor.on_trò|o>iain_batch_start/oversize_allocations.current‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955590144 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚146734926208512 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚148769846771712 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚611 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚81718333 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26086 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177046016 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚609 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚164288237266944 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚44928891 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚145547659895053 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2034902560256 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 _step‚144 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171º¯{ÿ2¼- _wandbruntime‚2081  _runtime‚2081f‚­ù÷&½ñM NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚36809340 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚22889951 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚166500822058496 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2028420793159 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚81762822 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31929 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26114 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214657 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚164377532547584 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚81788936 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚44953482 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚33598167 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411620 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚44979594 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917423104 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚166501068554752 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450980 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _step‚145 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899420160 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚36809340 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177046016 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚148849672460800 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚146814574507008 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2027523381539 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2036015376896 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955590144 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955590144 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚618 $ _timestamp‚1.788512455637678e+09 _runtime‚2081.109165813ÂD\Äü&¾öM MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26112 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚145627663101479 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 trainer/global_step‚1799 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚143599242308320 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚146814556504064 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2036015376896 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2035115956736 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 246496256 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚146814574507008 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177046016 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚81762822 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚33598787 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚36809342 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚148850589883904 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917423104 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚44953482 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚620 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2123289510912 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚146814556504064 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 398533632 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚164377556487680 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 222556160 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1179 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31929 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1204 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10708836 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899420160 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31684 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214657 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚164377532547584 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚81788936 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚44953482 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚33598167 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411620 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚44979594 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917423104 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚166501068554752 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450980 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _step‚145 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899420160 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚36809340 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177046016 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚148849672460800 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚146814574507008 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2027523381539 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2036015376896 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955590144 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955590144 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚618 $ _timestamp‚1.788512455637678e+09 _runtime‚2081.109165813 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚36809340 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚22889951 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚166500822058496 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2028420793159 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚81762822 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31929 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26114 EDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚614 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917409792 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2089657733632 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31684 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177046016 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2180482064384 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31929 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚23496373 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2180692050944 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚149503025709333 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31929 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚152806098719744 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚34488636 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 398533632 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 _runtime‚2136.891349212 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26088 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚37784342 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚34488022 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10992261 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚83946135 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984B¬×Šó'ÏíO D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1179 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 trainer/global_step‚1849 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26086 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214657 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 _step‚148 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917409792 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚150716440986112 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 209986560 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2090557140480 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2089657733632 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚37784344 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚46187879 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411412 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚46187879 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2090557140480 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚170934656428032 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚612 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚83946135 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955590144 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 233926656 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚614 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917409792 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2089657733632 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31684 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177046016 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2180482064384 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31929 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚23496373 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2180692050944 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚149503025709333 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31929 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚152806098719744 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚34488636 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 398533632 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 _runtime‚2136.891349212 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26088 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚37784342 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚34488022 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10992261 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 IDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214657 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚37804240 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2181864743936 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2082970038647 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007233 _runtime‚2138.004381819 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚150796089284608 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31684 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚168843517524480 trainer/global_step‚1849 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955590144 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 254884864 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2083867450267 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚150796089284608 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚168843493584384 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚23508749 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚46212496 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917423104 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10997359 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26112 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955590144 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177046016 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚34506108 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚171025382268416 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚83990624 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2090770536960 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚37804242 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26112 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚606 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 _step‚149 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚84016738 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚34506716 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚23508751 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚149582114464987 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚147498247014720 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚150796071281664 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚152887759241728 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31929 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26114 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚84016738 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411620 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899420160 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10997965 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚46186384 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚37804240 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚83990624 ó4Ò÷M _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚168843493584384 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚23508749 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚46212496 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917423104 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10997359 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26112 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955590144 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177046016 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚34506108 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚171025382268416 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.curr£>^‹Éent‚ 1182793728 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚83990624 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2090770536960 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚37804242 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26112 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚606 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 _step‚149 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚84016738 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚34506716 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚23508751 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚149582114464987 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚147498247014720 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚150796071281664 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚152887759241728 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31929 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26114 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚84016738 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411620 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899420160 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10997965 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚46186384 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚37804240 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚83990624 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚149581200014007 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚37804242 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚150796071281664 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚46212496 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214657 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚37804240 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2181864743936 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2082970038647 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007233 _runtime‚2138.004381819 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚150796089284608 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31684 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚168843517524480 trainer/global_step‚1849 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955590144 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 254884864 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2083867450267 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚150796089284608 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607•*Ï‹2Ó- _wandbruntime‚2138  _runtime‚2138áb–YXÔÒ /loss/FRt1/assignment_loss‚4.176234275102615 .loss/FRt3/assignment_loss‚2.95096752096395 0loss/FRt4/assignment_loss‚3.8263150691986083 2loss/SRqqt4/assignment_loss‚1.8894394047749348 1loss/FBt2/assignment_loss‚0.07817477294042911 _step‚150 1loss/FBt3/assignment_loss‚0.08116065547801554 /loss/FBt2/detection_loss‚0.7288833309089144 1loss/SRqqt1/assignment_loss‚2.127362335799262 2loss/SRqqt3/assignment_loss‚1.7665117261058185 %loss/total_loss‚44.67263273424575 trainer/global_step‚1849 % _timestamp‚1.7885125125353725e+09 0loss/FRt2/assignment_loss‚1.6198223194350367 /loss/FRt3/detection_loss‚2.1352608892761293 .loss/FRt4/detection_loss‚2.827558974424998 .loss/FBt1/detection_loss‚2.172367139296098 /loss/FRt2/detection_loss‚1.7097298049408456 .loss/FBt4/detection_loss‚2.208870995044708 epoch‚1 _runtime‚2138.005646295 1loss/SRqqt1/detection_loss‚2.3792355125769973 0loss/SRqqt2/detection_loss‚1.208904706686735 1loss/SRqqt3/detection_loss‚1.6617383933626115 2loss/SRqqt2/assignment_loss‚0.9966942555271089 1loss/FBt1/assignment_loss‚0.14177512504354375 0loss/FBt4/assignment_loss‚0.1606545347101928 /loss/FRt1/detection_loss‚2.1044263781772767 0loss/SRqqt4/detection_loss‚2.413580678976499 .loss/FBt3/detection_loss‚3.306963935494423 åp;]Õ× 1loss/SRqqt1/detection_loss‚2.3792355125769973 0loss/SRqqt2/detection_loss‚1.208904706686735 1loss/SRqqt3/detection_loss‚1.6617383933626115 2loss/SRqqt2/assignment_loss‚0.9966942555271089 1loss/FBt1/assignment_loss‚0.14177512504354375 0loss/FBt4/assignment_loss‚0.1606545347101928 /loss/FRt1/detection_loss‚2.1044263781772767 0loss/SRqqt4/detection_loss‚2.413580678976499 .loss/FBt3/detection_loss‚3.306963935494423 /loss/FRt1/assignment_loss‚4.176234275102615 .loss/FRt3/assignment_loss‚2.95096752096395 0loss/FRt4/assignment_loss‚3.8263150691986083 2loss/SRqqt4/assignment_loss‚1.8894394047749348 1loss/FBt2/assignment_loss‚0.07817477294042911 _step‚150 1loss/FBt3/assignment_loss‚0.08116065547801554 /loss/FBt2/detection_loss‚0.7288833309089144 1loss/SRqqt1/assignment_loss‚2.127362335799262 2loss/SRqqt3/assignment_loss‚1.7665117261058185 %loss/total_loss‚44.67263273424575 trainer/global_step‚1849 % _timestamp‚1.7885125125353725e+09 0loss/FRt2/assignment_loss‚1.6198223194350367 /loss/FRt3/detection_loss‚2.1352608892761293 .loss/FRt4/detection_loss‚2.827558974424998 .loss/FBt1/detection_loss‚2.172367139296098 /loss/FRt2/detection_loss‚1.7097298049408456 .loss/FBt4/detection_loss‚2.208870995044708 epoch‚1 _runtime‚2138.005646295–g{ª’Ö‚(:‡ ‡’êÔŠÝÐþ gpu.0.gpu‚41.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚131.45' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚32.8625 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ôˆŸÂSׂ(:È ‡’êÔ®ˆÿ disk.sda.in‚ 0.0390625 disk.rbd0.out‚ 0.279296875# disk.nvme1n1.out‚11469.91015625 disk.sdb1.out‚0 disk.sda1.out‚0 proc.cpu.threads‚17) disk./.usagePercent‚75.81086589665158 disk.rbd0.in‚0 disk.sdb1.in‚0 disk.md0.in‚ 0.0703125 disk.rbd2.in‚0 disk.nvme0n1.in‚ 0.02734375 network.sent‚ 3.409894e+06% disk./.usageGB‚2711.3637733459473 disk.sda2.in‚ 0.0390625 disk.sda2.out‚ 245.076171875 disk.dm-0.out‚23206.55078125 disk.md0p1.out‚ 236.1015625# memory_percent‚99.9751329421997& proc.memory.availableMB‚ 4.07421875 network.recv‚734928 disk.sdb.in‚0.03125 disk.sdb2.in‚0.03125 disk.sda1.in‚0* proc.memory.percent‚18.288755416870117 disk.md0.out‚ 236.1015625 disk.sdb2.out‚ 245.076171875 disk.md0p1.in‚ 0.0703125 disk.sda.out‚ 245.076171875 disk.rbd2.out‚ 0.50390625 disk.dm-0.in‚0.078125! disk.nvme0n1.out‚ 11736.640625" proc.memory.rssMB‚ 2996.4296875 disk.sdb.out‚ 245.076171875 disk.nvme1n1.in‚ 0.05078125 cpu‚25.171917330881683‚ե‘Ø‚(:† –’êÔ¨šÔÿ gpu.0.gpu‚36.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚37.0 gpu.0.powerWatts‚114.74' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.685 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Xø]Ò>Ù‚(:³ –’êÔɉ’€# disk.nvme0n1.out‚11813.98046875 disk.sdb1.out‚0 disk.sdb2.in‚0.03125 disk.sdb2.out‚246.5625 disk.md0p1.in‚ 0.0703125 disk.nvme0n1.in‚ 0.02734375 disk.sda2.out‚246.5625$ memory_percent‚99.97429847717285 proc.cpu.threads‚17 disk.sda2.in‚ 0.0390625 disk.rbd0.in‚0 disk.nvme1n1.in‚ 0.05078125 disk.sdb1.in‚0 disk.md0p1.out‚ 237.51953125" proc.memory.rssMB‚ 2996.4296875 disk.dm-0.in‚0.078125 disk.sdb.out‚246.5625 cpu‚25.171054662270805 network.recv‚741186# disk.nvme1n1.out‚11470.20703125 disk.md0.out‚ 237.51953125 disk.rbd2.out‚ 0.50390625 disk.sda1.in‚0 disk.dm-0.out‚ 23284.1875% proc.memory.availableMB‚ 4.2109375) disk./.usagePercent‚75.81297073559247 disk.sda.out‚246.5625 disk.rbd0.out‚ 0.279296875 disk.rbd2.in‚0 disk.sda1.out‚0 network.sent‚ 3.517993e+06 disk.sda.in‚ 0.0390625 disk.md0.in‚ 0.0703125 disk.sdb.in‚0.03125$ disk./.usageGB‚2711.439052581787* proc.memory.percent‚18.288755416870117Ààõ’Ú‚(:‡ ¥’êÔ·Õíý gpu.0.gpu‚16.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚177.688' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚44.422 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0®†¨JÛ‚(:¿ ¥’êÔÂú¤) disk./.usagePercent‚75.81390870831194 disk.sdb1.out‚0 disk.rbd2.out‚ 0.50390625 disk.rbd0.out‚ 0.279296875" proc.memory.rssMB‚ 2996.4296875 disk.sda1.out‚0 disk.md0p1.out‚ 239.015625* proc.memory.percent‚18.288755416870117 disk.sda2.in‚ 0.0390625 network.recv‚744907 disk.md0.out‚ 239.015625 disk.sdb2.out‚ 248.11328125 disk.dm-0.in‚0.078125 disk.sda.in‚ 0.0390625 disk.rbd0.in‚0 disk.sda.out‚ 248.11328125 disk.sda2.out‚ 248.11328125$ disk./.usageGB‚2711.472599029541 disk.nvme1n1.in‚ 0.05078125 disk.md0.in‚ 0.0703125 disk.sdb2.in‚0.03125 disk.sda1.in‚0& proc.memory.availableMB‚ 4.21484375 disk.sdb.in‚0.03125 disk.sdb.out‚ 248.11328125$ memory_percent‚99.97427463531494 cpu‚25.170089778939317 proc.cpu.threads‚17" disk.nvme1n1.out‚ 11470.3671875 network.sent‚ 3.525479e+06 disk.sdb1.in‚0" disk.nvme0n1.out‚ 12145.2265625 disk.nvme0n1.in‚ 0.02734375 disk.rbd2.in‚0 disk.dm-0.out‚ 23615.59375 disk.md0p1.in‚ 0.0703125ö%=(“Ü‚(:ˆ ´’êÔÜÔÿ gpu.0.gpu‚2.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚36.0 gpu.0.powerWatts‚114.131' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.53275 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0†4õôVÝ‚(:Ë ´’êԆ䔀& proc.memory.availableMB‚ 4.09765625 disk.nvme0n1.in‚ 0.02734375 disk.sdb.out‚250.13720703125$ memory_percent‚99.97498989105225 cpu‚25.16907641895937 disk.sdb.in‚0.03125 disk.sda.in‚ 0.0390625 disk.rbd0.out‚ 0.279296875 disk.dm-0.out‚23757.86328125 network.sent‚ 3.529278e+06 disk.md0p1.out‚ 240.96875 disk.sdb2.in‚0.03125 disk.md0p1.in‚ 0.0703125! disk.sda2.out‚250.13720703125 disk.sdb1.in‚0 disk.md0.in‚ 0.0703125 disk.md0.out‚ 240.96875 network.recv‚749219 disk.sda.out‚250.13720703125 disk.rbd2.in‚0! disk.sdb2.out‚250.13720703125" proc.memory.rssMB‚ 2996.4296875) disk./.usagePercent‚75.82003944903776$ disk./.usageGB‚2711.691864013672 disk.rbd0.in‚0 disk.nvme1n1.in‚ 0.05078125* proc.memory.percent‚18.288755416870117# disk.nvme1n1.out‚11470.66015625 disk.sdb1.out‚0 disk.sda1.in‚0 disk.dm-0.in‚0.078125 proc.cpu.threads‚17 disk.sda1.out‚0! disk.nvme0n1.out‚ 12287.203125 disk.sda2.in‚ 0.0390625 disk.rbd2.out‚ 0.50390625Ù™3k2Þ- _wandbruntime‚2193  _runtime‚2193[sàÇßÁ _step‚151 & lr-AdamW/pg1‚0.0008492185823334416 & lr-AdamW/pg2‚0.0008492185823334416 trainer/global_step‚1899 % _timestamp‚1.7885125681037874e+09 _runtime‚2193.574170939…ÄÓoÌàÆ & lr-AdamW/pg1‚0.0008492185823334416 & lr-AdamW/pg2‚0.0008492185823334416 trainer/global_step‚1899 % _timestamp‚1.7885125681037874e+09 _runtime‚2193.574170939 _step‚151—úW¢2á- _wandbruntime‚2193  _runtime‚2193ôKwÚî'âèO \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚154697955763712 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1179 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚175458903618560 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚154697973766656 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚38779242 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2238982398976 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚24115173 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚156843268077568 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2145312313856 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26088 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚47394695 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚35395493 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31684 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚156843268077568 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2137308034437 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚623 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 _runtime‚2193.574991803 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚154697955763712 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚24115175 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚153457477072841 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _step‚152 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2138205445849 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955590144 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 238120960 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚38779242 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26086 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚38779244 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11280318 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26088 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11279697 trainer/global_step‚1899 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚621 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚173220135400448 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2145312313856 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚47394695 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 214180864 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚153456562622069 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214657™œAtãíO P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955590144 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 238120960 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚38779242 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26086 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚38779244 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11280318 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26088 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11279697 trainer/global_step‚1899 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚621 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚173220135400448 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2145312313856 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚47394695 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 214180864 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚153456562622069 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214657 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚154697955763712 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1179 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚175458903618560 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚154697973766656 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚38779242 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2238982398976 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚24115173 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚156843268077568 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2145312313856 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26088 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚47394695 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚35395493 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31684 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚156843268077568 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2137308034437 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚623 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 _runtime‚2193.574991803 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚154697955763712 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚24115175 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚153457477072841 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _step‚152 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2138205445849 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0˜Vý„2ä- _wandbruntime‚2194  _runtime‚2194ÒTÞø&åòM VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899420160 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177046016 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚86218426 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚611 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚154777604062208 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚175549630067200 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚151397251721120 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 % _timestamp‚1.7885125691888244e+09 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚173309478561280 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚173309454621184 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _runtime‚2194.661040581 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214657 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚24127551 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2139314107375 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚154777604062208 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177046016 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚47419286 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2146425117184 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚175549385668096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411620 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1204 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚47445398 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚47445398 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26112 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚86244540 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31684 trainer/global_step‚1899 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 398533632 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚38799140 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚35413005 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚153535651377515 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917423104 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31929 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11285456 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚38799142 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1179 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899420160 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007233 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955590144 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1547775860592644fý&æ÷M KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚47445398 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26112 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚86244540 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31684 trainer/global_step‚1899 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 398533632 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚38799140 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚35413005 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚153535651377515 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917423104 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31929 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11285456 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚38799142 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1179 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899420160 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007233 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955590144 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚154777586059264 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899420160 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177046016 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚86218426 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚611 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚154777604062208 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚175549630067200 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚151397251721120 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 % _timestamp‚1.7885125691888244e+09 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚173309478561280 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚173309454621184 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _runtime‚2194.661040581 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214657 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚24127551 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2139314107375 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚154777604062208 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177046016 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚47419286 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2146425117184 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚175549385668096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411620 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1204 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚47445398™"ŠH2ç- _wandbruntime‚2194  _runtime‚2194ÚUZèÔ /loss/FBt4/detection_loss‚2.8686864574750266 1loss/FBt3/assignment_loss‚0.16020087763899937 .loss/FRt3/detection_loss‚2.222976523141066 0loss/SRqqt4/detection_loss‚2.722605940596811 /loss/FBt2/detection_loss‚0.5974730267439798 0loss/SRqqt2/detection_loss‚1.420271126655015 _runtime‚2194.662284917 /loss/FRt4/assignment_loss‚4.090486838537104 2loss/SRqqt2/assignment_loss‚1.4132882866669785 1loss/FBt1/assignment_loss‚0.04834371858409473 1loss/SRqqt1/detection_loss‚1.8804203191628823 epoch‚1 $ _timestamp‚1.788512569191984e+09 0loss/FRt2/assignment_loss‚1.5768370471502606 2loss/SRqqt1/assignment_loss‚1.5424997984216764 2loss/SRqqt3/assignment_loss‚1.6464509692043066 2loss/SRqqt4/assignment_loss‚1.8795422369550014 1loss/FBt2/assignment_loss‚0.10742820180856354 .loss/FRt2/detection_loss‚1.929992242079032 .loss/FRt4/detection_loss‚2.848746098139707 /loss/FBt1/detection_loss‚2.3517500332423618 0loss/FRt1/assignment_loss‚2.5991813273265443 0loss/FBt4/assignment_loss‚0.2860133186914027 0loss/SRqqt3/detection_loss‚1.697567651172479 %loss/total_loss‚44.05081149444098 trainer/global_step‚1899 0loss/FRt3/assignment_loss‚3.0650715343654156 /loss/FRt1/detection_loss‚1.9107815155695225 _step‚154 /loss/FBt3/detection_loss‚3.1841964051127434c˜Ö_éÙ 0loss/SRqqt4/detection_loss‚2.722605940596811 /loss/FBt2/detection_loss‚0.5974730267439798 0loss/SRqqt2/detection_loss‚1.420271126655015 _runtime‚2194.662284917 /loss/FRt4/assignment_loss‚4.090486838537104 2loss/SRqqt2/assignment_loss‚1.4132882866669785 1loss/FBt1/assignment_loss‚0.04834371858409473 1loss/SRqqt1/detection_loss‚1.8804203191628823 epoch‚1 $ _timestamp‚1.788512569191984e+09 0loss/FRt2/assignment_loss‚1.5768370471502606 2loss/SRqqt1/assignment_loss‚1.5424997984216764 2loss/SRqqt3/assignment_loss‚1.6464509692043066 2loss/SRqqt4/assignment_loss‚1.8795422369550014 1loss/FBt2/assignment_loss‚0.10742820180856354 .loss/FRt2/detection_loss‚1.929992242079032 .loss/FRt4/detection_loss‚2.848746098139707 /loss/FBt1/detection_loss‚2.3517500332423618 0loss/FRt1/assignment_loss‚2.5991813273265443 0loss/FBt4/assignment_loss‚0.2860133186914027 0loss/SRqqt3/detection_loss‚1.697567651172479 %loss/total_loss‚44.05081149444098 trainer/global_step‚1899 0loss/FRt3/assignment_loss‚3.0650715343654156 /loss/FRt1/detection_loss‚1.9107815155695225 _step‚154 /loss/FBt3/detection_loss‚3.1841964051127434 /loss/FBt4/detection_loss‚2.8686864574750266 1loss/FBt3/assignment_loss‚0.16020087763899937 .loss/FRt3/detection_loss‚2.222976523141066š”ò>R”ê‚(:‰ Ã’êԎ⌀ gpu.0.gpu‚55.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚174.903' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.72575 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0°*Më‚(: ÒêÔ€ disk.nvme1n1.in‚ 0.05078125 disk.sdb1.in‚0 disk.md0.in‚ 0.0703125 disk.rbd2.in‚0 disk.sda1.in‚0 disk.md0p1.out‚ 242.5390625 disk.nvme0n1.in‚ 0.02734375" proc.memory.rssMB‚ 2996.4296875 disk.sda2.out‚251.7724609375 disk.rbd2.out‚ 0.50390625 disk.sda1.out‚0 disk.dm-0.in‚0.078125# memory_percent‚99.9729871749878 disk.sdb2.out‚251.7724609375 disk.dm-0.out‚ 23960.515625 disk.sda.out‚251.7724609375 disk.sda2.in‚ 0.0390625 disk.rbd0.in‚0 proc.cpu.threads‚17 disk.rbd0.out‚ 0.279296875 disk.nvme1n1.out‚ 11470.78125) disk./.usagePercent‚75.82002664977391 disk./.usageGB‚ 2711.69140625 disk.md0p1.in‚ 0.0703125 disk.sdb.in‚0.03125 network.sent‚ 3.535086e+06 network.recv‚753274 disk.md0.out‚ 242.5390625! disk.nvme0n1.out‚ 12489.734375 disk.sdb1.out‚0* proc.memory.percent‚18.288755416870117 disk.sdb2.in‚0.03125 disk.sdb.out‚251.7724609375& proc.memory.availableMB‚ 4.42578125 cpu‚25.168268380328115 disk.sda.in‚ 0.0390625™F“ì‚(:ˆ Ò’êÔŸª¾ÿ gpu.0.gpu‚57.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚174.562' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.6405 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ïUWóYí‚(:Î Ò’êÔ¼££€ disk.sda2.in‚ 0.0390625& proc.memory.availableMB‚ 3.91015625 network.recv‚758417 disk.rbd0.in‚0 disk.rbd0.out‚ 0.279296875 disk.dm-0.in‚0.078125! disk.nvme0n1.out‚ 12689.484375 cpu‚25.167260210695062) disk./.usagePercent‚75.81147812810596$ disk./.usageGB‚2711.385669708252 disk.md0.in‚ 0.0703125 disk.md0.out‚ 244.203125 disk.sdb2.in‚0.03125 disk.sda1.in‚0 disk.nvme0n1.in‚ 0.02734375$ memory_percent‚99.97613430023193 disk.nvme1n1.in‚ 0.05078125 disk.rbd2.in‚0! disk.sdb2.out‚253.50732421875 disk.sda1.out‚0 disk.dm-0.out‚24160.51953125 disk.sdb.in‚0.03125" proc.memory.rssMB‚ 2996.4296875 disk.rbd2.out‚ 0.50390625 disk.md0p1.in‚ 0.0703125 disk.sdb.out‚253.50732421875 disk.sda.in‚ 0.0390625! disk.sda2.out‚253.50732421875* proc.memory.percent‚18.288755416870117 proc.cpu.threads‚17 network.sent‚ 3.641994e+06# disk.nvme1n1.out‚11471.03515625 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.md0p1.out‚ 244.203125 disk.sda.out‚253.50732421875ºv •“î‚(:ˆ á’êÔÆÕ¹ÿ gpu.0.gpu‚56.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚39.0 gpu.0.powerWatts‚174.562' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.6405 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0·³Q"Eï‚(:º á’êÔЗ€$ disk./.usageGB‚2711.385757446289 disk.rbd2.in‚0 disk.sdb2.out‚ 255.4609375 network.recv‚761807 disk.sda1.in‚0 disk.sda.in‚ 0.0390625 disk.nvme1n1.in‚ 0.0546875 disk.rbd0.in‚0 disk.sdb2.in‚0.03125 cpu‚25.166262841339652 disk.sdb1.in‚0 disk.rbd2.out‚ 0.50390625 disk.dm-0.out‚ 24160.78125 disk.md0p1.in‚ 0.0703125 disk.sda2.in‚ 0.0390625 disk.sda2.out‚ 255.4609375! disk.nvme1n1.out‚ 11471.234375% proc.memory.availableMB‚ 4.5234375 network.sent‚ 3.649397e+06! disk.nvme0n1.out‚ 12689.546875 disk.md0.in‚ 0.0703125 disk.nvme0n1.in‚ 0.02734375 disk.rbd0.out‚ 0.279296875 disk.sdb1.out‚0 proc.cpu.threads‚17( disk./.usagePercent‚75.8114805812982 disk.sda1.out‚0 disk.sdb.in‚0.03125 disk.sdb.out‚ 255.4609375 disk.md0.out‚ 246.0859375$ memory_percent‚99.97239112854004" proc.memory.rssMB‚ 2996.4296875* proc.memory.percent‚18.288755416870117 disk.dm-0.in‚ 0.08203125 disk.md0p1.out‚ 246.0859375 disk.sda.out‚ 255.4609375Óh)Ú2ð- _wandbruntime‚2247  _runtime‚2247Ö2qaÇñÁ & lr-AdamW/pg1‚0.0008479496131817339 & lr-AdamW/pg2‚0.0008479496131817339 trainer/global_step‚1949 % _timestamp‚1.7885126215570738e+09 _runtime‚2247.027415717 _step‚155imTèÌòÆ trainer/global_step‚1949 % _timestamp‚1.7885126215570738e+09 _runtime‚2247.027415717 _step‚155 & lr-AdamW/pg1‚0.0008479496131817339 & lr-AdamW/pg2‚0.0008479496131817339›§°2ó- _wandbruntime‚2247  _runtime‚2247__; UôèO _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2297035344896 _runtime‚2247.028637173 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚36303003 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917409792 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2200966894080 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚48627597 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2193652102957 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899406848 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2200966894080 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31684 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1204 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411412 F@DeviceStatsMonitor.on_train_batch_start/reser×c•6™!ved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚160880437435392 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚24733975 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450772 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26088 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚158679470541312 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚604 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 224666624 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚88401739 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚39774142 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚160881354845184 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177046016 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚155218259294032 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26086 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1179 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 248606720 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26086 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007233 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917409792 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 trainer/global_step‚1949 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899406848 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚88427827 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚39774144 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚39774142 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚177686096437248 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 % _timestamp‚1.7885126215582232e+09 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚48653683 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955590144 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚602 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚157411013985577 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2192754691545 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214657 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31684 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955590144 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11569632 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚155218276333392 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚158679470541312 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2201866300928 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26088 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚24733973 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11569030 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 398533632 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚39774144 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚48653683 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _step‚156 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚158679488544256 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚179983380388864 ØöZó'õíO L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚48627597 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31684 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955590144 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11569632 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚155218276333392 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚158679470541312 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2201866300928 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26088 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚24733973 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11569030 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 398533632 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚39774144 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚48653683 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _step‚156 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚158679488544256 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚179983380388864 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2297035344896 _runtime‚2247.028637173 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚36303003 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917409792 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2200966894080 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚48627597 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2193652102957 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899406848 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2200966894080 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31684 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1204 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411412 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚160880437435392 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚24733975 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450772 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26088 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚158679470541312 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚604 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 224666624 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚88401739 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚39774142 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚160881354845184 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177046016 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚155218259294032 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26086 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1179 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 248606720 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26086 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007233 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917409792 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 trainer/global_step‚1949 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899406848 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚88427827 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚39774144 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚39774142 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚177686096437248 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 % _timestamp‚1.7885126215582232e+09 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚48653683 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955590144 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚602 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚157411013985577 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2192754691545 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214657 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 IDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚88472342 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚36321067 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411620 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31929 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 _runtime‚2247.973454504 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26114 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11574718 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚88446228 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚158759100836864 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚48678300 _step‚157 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917423104 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚158759100836864 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚160961180534272 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2202979117568 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 236010496 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚39794042 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚155296239388160 % _timestamp‚1.7885126225023773e+09 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚160962097957376 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚39794042 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚24746351 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955590144 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2202979117568 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚610 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1179 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚160962097957376 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 212070400 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31929 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚39794040 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2194760764483 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177046016 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2193863352863 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚39794040 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26112 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚24746349 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214657 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚158759118839808 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚177775439598080 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450980 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2202079697408 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2202079697408 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007233 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚36321677 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚608 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚155296256427520 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31684 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 398533632 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚177775415657984 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚157490102741023 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 Fwˆ´'ø÷M UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2202079697408 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2202079697408 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007233 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚36321677 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚608 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚155296256427520 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31684 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 398533632 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚177775415657984 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚157490102741023 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚88472342 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚36321067 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411620 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31929 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 _runtime‚2247.973454504 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26114 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11574718 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚88446228 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚158759100836864 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚48678300 _step‚157 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917423104 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚158759100836864 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚160961180534272 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2202979117568 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 236010496 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚39794042 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚155296239388160 % _timestamp‚1.7885126225023773e+09 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚160962097957376 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚39794042 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚24746351 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955590144 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2202979117568 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚610 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1179 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚160962097957376 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 212070400 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31929 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚39794040 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2194760764483 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177046016 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2193863352863 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚39794040 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26112 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚24746349 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214657 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚158759118839808 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚177775439598080 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450980Þ*–ô2ù- _wandbruntime‚2247  _runtime‚2247›(bÏWúÑ /loss/FRt1/detection_loss‚2.2189592147360044 %loss/total_loss‚42.51693789477697 trainer/global_step‚1949 /loss/FRt2/assignment_loss‚1.702276087084482 1loss/FBt1/assignment_loss‚0.17228676083427022 1loss/SRqqt2/detection_loss‚1.1441956651547263 /loss/FBt1/detection_loss‚2.2281278712408885 /loss/FBt2/detection_loss‚0.7307474441826344 /loss/FRt3/assignment_loss‚2.896156627602047 .loss/FBt3/detection_loss‚3.120873846113682 2loss/SRqqt1/assignment_loss‚2.0769225111667144 1loss/FBt2/assignment_loss‚0.10183632816383476 .loss/FRt2/detection_loss‚1.814137416870095 /loss/FBt4/detection_loss‚3.2413813405566745 _step‚158 0loss/FRt1/assignment_loss‚2.7607366904920463 0loss/FBt4/assignment_loss‚0.1744648876806928 2loss/SRqqt3/assignment_loss‚1.5835453624216218 0loss/SRqqt4/assignment_loss‚1.88504451148364 .loss/FRt4/detection_loss‚2.581690397527483 _runtime‚2247.975347493 /loss/FRt4/assignment_loss‚2.634079376856486 2loss/SRqqt2/assignment_loss‚0.7287241181944527 1loss/FBt3/assignment_loss‚0.16880142618902028 epoch‚1 .loss/FRt3/detection_loss‚2.422719054089652 1loss/SRqqt1/detection_loss‚2.0483603336392564 0loss/SRqqt3/detection_loss‚2.034881855982045 0loss/SRqqt4/detection_loss‚2.045988766514525 % _timestamp‚1.7885126225051563e+09”‚ëÀ\ûÖ %loss/total_loss‚42.51693789477697 trainer/global_step‚1949 /loss/FRt2/assignment_loss‚1.702276087084482 1loss/FBt1/assignment_loss‚0.17228676083427022 1loss/SRqqt2/detection_loss‚1.1441956651547263 /loss/FBt1/detection_loss‚2.2281278712408885 /loss/FBt2/detection_loss‚0.7307474441826344 /loss/FRt3/assignment_loss‚2.896156627602047 .loss/FBt3/detection_loss‚3.120873846113682 2loss/SRqqt1/assignment_loss‚2.0769225111667144 1loss/FBt2/assignment_loss‚0.10183632816383476 .loss/FRt2/detection_loss‚1.814137416870095 /loss/FBt4/detection_loss‚3.2413813405566745 _step‚158 0loss/FRt1/assignment_loss‚2.7607366904920463 0loss/FBt4/assignment_loss‚0.1744648876806928 2loss/SRqqt3/assignment_loss‚1.5835453624216218 0loss/SRqqt4/assignment_loss‚1.88504451148364 .loss/FRt4/detection_loss‚2.581690397527483 _runtime‚2247.975347493 /loss/FRt4/assignment_loss‚2.634079376856486 2loss/SRqqt2/assignment_loss‚0.7287241181944527 1loss/FBt3/assignment_loss‚0.16880142618902028 epoch‚1 .loss/FRt3/detection_loss‚2.422719054089652 1loss/SRqqt1/detection_loss‚2.0483603336392564 0loss/SRqqt3/detection_loss‚2.034881855982045 0loss/SRqqt4/detection_loss‚2.045988766514525 % _timestamp‚1.7885126225051563e+09 /loss/FRt1/detection_loss‚2.2189592147360044ž»`_”ü‚(:‰ ð’êÔ¥ò²ÿ gpu.0.gpu‚58.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚37.0 gpu.0.powerWatts‚116.583' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚29.14575 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ÿ÷ZVý‚(:Ë ð’êÔÉãæÿ disk.dm-0.in‚ 0.08203125$ memory_percent‚99.97649192810059% proc.memory.availableMB‚ 3.8515625 disk.sdb.out‚257.0556640625 disk.nvme0n1.in‚ 0.02734375# disk.nvme0n1.out‚12922.02734375 cpu‚25.165395349824315 disk.sdb.in‚0.03125 disk.sda.out‚257.0556640625 disk.md0.in‚ 0.0703125 disk.sda.in‚ 0.0390625 disk.sda2.in‚ 0.0390625 network.sent‚ 3.66604e+06% disk./.usageGB‚2711.2992095947266 disk.rbd0.in‚0 disk.dm-0.out‚ 24393.71875 disk.nvme1n1.in‚ 0.0546875 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.sda1.in‚0 disk.sda2.out‚257.0556640625 disk.md0p1.in‚ 0.0703125 disk.md0p1.out‚ 247.59765625 disk.md0.out‚ 247.59765625 disk.sda1.out‚0* proc.memory.percent‚18.288755416870117 disk.sdb2.out‚257.0556640625" proc.memory.rssMB‚ 2996.4296875 proc.cpu.threads‚17 network.recv‚766297) disk./.usagePercent‚75.80906066714542 disk.rbd0.out‚ 0.357421875# disk.nvme1n1.out‚11471.69140625 disk.rbd2.in‚0 disk.rbd2.out‚0.515625 disk.sdb2.in‚0.03125Ø5&ì“þ‚(:ˆ ÿ’êÔº÷§ÿ gpu.0.gpu‚8.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚170.219' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.55475 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ç™îbYÿ‚(:Î ÿ’êÔ®ž‰€ disk.sdb1.in‚0 disk.md0.out‚ 249.08984375 disk.dm-0.out‚ 24532.8671875 disk.rbd0.out‚ 0.357421875 disk.dm-0.in‚ 0.08203125 disk.md0p1.in‚ 0.0703125 disk.nvme0n1.in‚ 0.02734375% disk./.usageGB‚2711.4376792907715 disk.rbd0.in‚0! disk.nvme1n1.out‚ 11542.734375 disk.md0.in‚ 0.0703125 disk.rbd2.out‚0.578125! disk.sdb2.out‚258.61279296875( disk./.usagePercent‚75.8129323378009 disk.sda.out‚258.61279296875 disk.sda2.in‚ 0.0390625 disk.nvme1n1.in‚ 0.0546875 disk.md0p1.out‚ 249.08984375* proc.memory.percent‚18.288755416870117 network.sent‚ 3.763184e+06 disk.sda.in‚ 0.0390625! disk.sda2.out‚258.61279296875 disk.rbd2.in‚0 disk.sdb.in‚0.03125 proc.cpu.threads‚17" disk.nvme0n1.out‚ 12990.1328125 disk.sdb.out‚258.61279296875& proc.memory.availableMB‚ 4.31640625" proc.memory.rssMB‚ 2996.4296875 network.recv‚772164 disk.sda1.in‚0 disk.sda1.out‚0 disk.sdb1.out‚0 disk.sdb2.in‚0.03125$ memory_percent‚99.97365474700928 cpu‚25.164415158745159]‘€‚(:† Ž“êÔá„–ÿ gpu.0.gpu‚14.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚37.0 gpu.0.powerWatts‚114.74' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.685 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0m÷g\‚(:Ñ Ž“êÔÝ™×ÿ disk.sdb1.in‚0 disk.dm-0.in‚ 0.08203125 disk.sdb.in‚0.03125( disk./.usagePercent‚75.8129323378009 disk.rbd2.in‚0 disk.sda1.in‚0 disk.md0p1.in‚ 0.0703125 disk.sda.out‚260.60400390625$ memory_percent‚99.97663497924805# disk.nvme1n1.out‚11542.98828125 disk.md0.out‚ 251.02734375! disk.sdb2.out‚260.60400390625 disk.nvme0n1.in‚ 0.02734375 cpu‚25.163571140322887 disk.nvme1n1.in‚ 0.0546875 disk.sda.in‚ 0.0390625% disk./.usageGB‚2711.4376792907715! disk.sda2.out‚260.60400390625 disk.rbd2.out‚0.578125 disk.sdb2.in‚0.03125 disk.sda1.out‚0 disk.md0p1.out‚ 251.02734375 network.sent‚ 3.76944e+06 disk.rbd0.in‚0 disk.sdb1.out‚0 disk.md0.in‚ 0.0703125" proc.memory.rssMB‚ 2996.4296875 disk.sda2.in‚ 0.0390625 disk.rbd0.out‚ 0.45947265625# disk.nvme0n1.out‚13037.75390625$ proc.memory.availableMB‚3.828125 network.recv‚776893 disk.dm-0.out‚ 24580.7421875 disk.sdb.out‚260.60400390625* proc.memory.percent‚18.288755416870117 proc.cpu.threads‚17æɈº’‚‚(:‡ “êÔœ¯“þ gpu.0.gpu‚56.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚37.0 gpu.0.powerWatts‚177.688' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚44.422 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ëMˆ=Wƒ‚(:Ì “êÔ·ä’ÿ disk.sdb1.out‚0 disk.sdb2.in‚0.03125 disk.sda.in‚ 0.0390625 network.recv‚780475 disk.sdb.in‚0.03125$ memory_percent‚99.97670650482178) disk./.usagePercent‚75.81298705465387 disk.rbd0.in‚0 disk.md0.in‚ 0.0703125 disk.sda1.in‚0 disk.dm-0.out‚24886.77734375 disk.md0p1.out‚ 252.609375 network.sent‚ 3.772536e+06 disk.rbd2.in‚0 disk.sdb2.out‚ 262.251953125 disk.dm-0.in‚ 0.08203125 disk.sda2.in‚ 0.0390625 disk.rbd0.out‚ 0.45947265625 disk.rbd2.out‚ 0.6572265625 disk.md0p1.in‚ 0.0703125 disk.sdb.out‚ 262.251953125# disk.nvme1n1.out‚11618.08203125 disk.sdb1.in‚0 disk.md0.out‚ 252.609375" proc.memory.rssMB‚ 2996.4296875* proc.memory.percent‚18.288755416870117% disk./.usageGB‚2711.4396362304688 disk.sda1.out‚0 disk.sda.out‚ 262.251953125& proc.memory.availableMB‚ 3.81640625 disk.nvme1n1.in‚ 0.0546875 disk.nvme0n1.in‚ 0.02734375" disk.nvme0n1.out‚ 13268.6953125 disk.sda2.out‚ 262.251953125 cpu‚25.16263362660384 proc.cpu.threads‚17$v/ÿ2„- _wandbruntime‚2299  _runtime‚2299gQ°•Ç…Á % _timestamp‚1.7885126744971797e+09 _runtime‚2299.967590018 _step‚159 & lr-AdamW/pg1‚0.0008465933043613623 & lr-AdamW/pg2‚0.0008465933043613623 trainer/global_step‚1999†‚‚&Ì†Æ _runtime‚2299.967590018 _step‚159 & lr-AdamW/pg1‚0.0008465933043613623 & lr-AdamW/pg2‚0.0008465933043613623 trainer/global_step‚1999 % _timestamp‚1.7885126744971797e+09ŸP¶µ2‡- _wandbruntime‚2299  _runtime‚2299,6Õî'ˆèO TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚25352773 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 _step‚160 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899406848 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚49886585 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31929 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177046016 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2248201348653 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2256621474304 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚40769042 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 246509568 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26088 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚164917606793216 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11857228 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2355361150464 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917409792 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26088 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚162661003321856 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚49860499 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2257520881152 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899406848 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2256621474304 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚37210003 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚611 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚184507442564608 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2249098760065 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955590144 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2257520881152 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1179 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚161365465349085 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26086 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚49886585 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚162660985318912 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31684 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚162661003321856 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚184507196055040 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26088 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚164917606793216 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11857228 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2355361150464 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917409792 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26088 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚162661003321856 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚49860499 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2257520881152 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899406848 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2256621474304 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚37210003 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚611 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚184507442564608 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2249098760065 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955590144 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 22575208811x*š½ 52 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1179 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚161365465349085 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26086 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚49886585 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚162660985318912 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31684 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚162661003321856 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚184507196055040 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 IDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31929 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚49885090 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955590144 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚90674030 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚49911202 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚164998349892096 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚90700144 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚40788942 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚37228065 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚182241400634880 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450980 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 _runtime‚2300.934089086 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚90674030 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚25365149 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 398533632 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚49911202 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚600 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 trainer/global_step‚1999 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚161445468555511 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚159195244094560 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚40788942 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚162740633617408 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11862314 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚184597934661632 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚184597696553984 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899420160 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2257734277632 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚182241376694784 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚40788940 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26112 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚162740615614464 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚25365151 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 238107648 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚161444554104531 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚159195261133920 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26114 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917423104 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚49885090 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2356534026752 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2250207421591 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚162740633617408 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚37227463 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 % _timestamp‚1.7885126754619713e+09 _step‚161 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚90700144 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214657 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955590144 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚90674030 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚49911202 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚164998349892096 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚90700144 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚40788942 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚37228065 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚182241400634880 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450980 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 _runtime‚2300.934089086 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚90674030 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚25365149 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 398533632 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚49911202 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚600 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 trainer/global_step‚1999 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚161445468555511 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚159195244094560 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚40788942 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚162740633617408 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11862314 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚184597934661632 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚184597696553984 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899420160 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2257734277632 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚182241376694784 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚40788940 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26112 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚162740615614464 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚25365151 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 238107648 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚161444554104531 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚159195261133920 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26114 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917423104 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚49885090 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2356534026752 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2250207421591 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚162740633617408 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚37227463 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 % _timestamp‚1.7885126754619713e+09 _step‚161 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚90700144 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214657 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31929 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚49885090 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 FDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚168954776151040 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411412 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚168954776151040 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31684 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚186618042450944 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚165320831163365 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚163016285746192 % _timestamp‚1.7885127314580834e+09 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917409792 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚189031695002112 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 398533632 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 _step‚164 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26086 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚168955693560832 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31929 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚41763942 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚163016268706832 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899406848 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚92883431 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26088 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚38116325 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚25971575 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚166642500096512 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚38116955 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 220472320 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0µ¾í.ó'íO YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31684 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚186618042450944 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚165320831163365 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚163016285746192 % _timestamp‚1.7885127314580834e+09 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917409792 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚189031695002112 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 398533632 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 _step‚164 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26086 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚168955693560832 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31929 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚41763942 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚163016268706832 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899406848 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚92883431 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26088 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚38116325 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚25971575 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚166642500096512 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚38116955 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 220472320 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2313175461376 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1179 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚189031450589696 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚41763944 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1204 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12145380 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚186618018510848 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31929 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚51093401 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31684 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚630 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚168954776151040 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411412 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚168954776151040¤vµ¢p2ž- _wandbruntime‚2357  _runtime‚2357Mqt§÷&ŸñM TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 252787712 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26112 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚92927946 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 228847616 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177046016 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚25983949 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚41783842 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2414808475136 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917423104 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚166722148395008 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899420160 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚25983951 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12150987 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2313388857856 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚41783840 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚163094265840320 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚628 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚165399919919019 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚51117992 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955590144 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚169036436673024 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚38134308 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2314288278016 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚186707361671680 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31684 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚189122170146816 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚51144104 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚186707337731584 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚169036436673024 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚92927946 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚51144104 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2313388857856 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 398533632 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450980 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚51117992 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31929 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚169035519249920 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚166722130392064 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1179 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚166722148395008 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚166722130392064 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚41783842 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214657 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚163094248800960 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917423104 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26114 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚41783840 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899420160 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177046016 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚38134938 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2414579627520 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 _runtime‚2357.964838153 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214657 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚163094248800960 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917423104 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26114 UHDeviceStatsMonitor.on_train_ba¡Œl%tch_end/allocated_bytes.large_pool.current‚18002944 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚41783840 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899420160 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177046016 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚38134938 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2414579627520 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 _runtime‚2357.964838153 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚92927946 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 228847616 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177046016 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚25983949 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚41783842 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2414808475136 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917423104 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚166722148395008 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899420160 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚25983951 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12150987 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2313388857856 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚41783840 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚163094265840320 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚628 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚165399919919019 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚51117992 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955590144 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚169036436673024 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚38134308 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2314288278016 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚186707361671680 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31684 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚189122170146816 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚51144104 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚186707337731584 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚169036436673024 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚92927946 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚51144104 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2313388857856 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 398533632 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450980 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728¥U{ž¹2¡- _wandbruntime‚2357  _runtime‚2357Éáb¢Ü .loss/FRt4/detection_loss‚2.750313762356253 .loss/FBt1/detection_loss‚3.446395585934321 /loss/FBt3/detection_loss‚2.7521081268787384 /loss/FBt4/detection_loss‚2.7103669233620167 _runtime‚2357.966514761 2loss/SRqqt1/assignment_loss‚1.4427739733866616 2loss/SRqqt2/assignment_loss‚1.2392466576311036 1loss/FBt2/assignment_loss‚0.07383896746547043 /loss/FRt2/detection_loss‚1.8218308481432142 1loss/SRqqt2/detection_loss‚1.2627468941652256 1loss/SRqqt4/detection_loss‚2.2500797066577647 2loss/SRqqt3/assignment_loss‚1.5455906968563795 2loss/SRqqt4/assignment_loss‚2.3199511184248816 epoch‚1 &loss/total_loss‚43.169344454753066 % _timestamp‚1.7885127324963784e+09 /loss/FRt3/assignment_loss‚2.688673587732537 0loss/FRt4/assignment_loss‚3.7291285045006695 1loss/FBt3/assignment_loss‚0.10323811191301502 /loss/FBt2/detection_loss‚0.7180845035725961 0loss/FRt1/assignment_loss‚3.5321217876799564 1loss/SRqqt1/detection_loss‚1.7268016276990665 0loss/FRt2/assignment_loss‚1.1324867407480876 2loss/FBt4/assignment_loss‚0.005975142703050551 /loss/FRt3/detection_loss‚2.2062016929997954 _step‚166 2loss/FBt1/assignment_loss‚0.019422206777865842 .loss/FRt1/detection_loss‚2.005540656599593 1loss/SRqqt3/detection_loss‚1.6864266305648048 trainer/global_step‚2049QÉz g£á 2loss/FBt4/assignment_loss‚0.005975142703050551 /loss/FRt3/detection_loss‚2.2062016929997954 _step‚166 2loss/FBt1/assignment_loss‚0.019422206777865842 .loss/FRt1/detection_loss‚2.005540656599593 1loss/SRqqt3/detection_loss‚1.6864266305648048 trainer/global_step‚2049 .loss/FRt4/detection_loss‚2.750313762356253 .loss/FBt1/detection_loss‚3.446395585934321 /loss/FBt3/detection_loss‚2.7521081268787384 /loss/FBt4/detection_loss‚2.7103669233620167 _runtime‚2357.966514761 2loss/SRqqt1/assignment_loss‚1.4427739733866616 2loss/SRqqt2/assignment_loss‚1.2392466576311036 1loss/FBt2/assignment_loss‚0.07383896746547043 /loss/FRt2/detection_loss‚1.8218308481432142 1loss/SRqqt2/detection_loss‚1.2627468941652256 1loss/SRqqt4/detection_loss‚2.2500797066577647 2loss/SRqqt3/assignment_loss‚1.5455906968563795 2loss/SRqqt4/assignment_loss‚2.3199511184248816 epoch‚1 &loss/total_loss‚43.169344454753066 % _timestamp‚1.7885127324963784e+09 /loss/FRt3/assignment_loss‚2.688673587732537 0loss/FRt4/assignment_loss‚3.7291285045006695 1loss/FBt3/assignment_loss‚0.10323811191301502 /loss/FBt2/detection_loss‚0.7180845035725961 0loss/FRt1/assignment_loss‚3.5321217876799564 1loss/SRqqt1/detection_loss‚1.7268016276990665 0loss/FRt2/assignment_loss‚1.1324867407480876¦ySå7”¤‚(:‰ è“êÔÖ²€ gpu.0.gpu‚41.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚157.243' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚39.31075 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0p€.]¥‚(:Ò è“êÔüö† disk.md0p1.in‚ 0.0703125 disk.sdb.in‚0.03125 disk.nvme0n1.in‚ 0.02734375 disk.sda.out‚270.56787109375! disk.sda2.out‚270.56787109375 disk.sdb1.out‚0 disk.sdb.out‚270.56787109375$ memory_percent‚99.97437000274658 proc.cpu.threads‚17) disk./.usagePercent‚75.81299494753326 disk.sdb1.in‚0 disk.rbd2.out‚ 0.783203125 disk.sda1.in‚0 disk.sda2.in‚ 0.0390625 disk.rbd0.out‚ 0.63330078125% disk./.usageGB‚2711.4399185180664! disk.nvme1n1.out‚ 12196.453125 disk.md0.out‚ 260.60546875 disk.sdb2.in‚0.03125 disk.dm-0.in‚ 0.08203125 disk.dm-0.out‚ 25630.296875 disk.md0p1.out‚ 260.60546875 disk.nvme0n1.out‚ 13433.84375 network.sent‚ 3.903717e+06 network.recv‚802988 disk.md0.in‚ 0.0703125 disk.sda1.out‚0 disk.sda.in‚ 0.0390625& proc.memory.availableMB‚ 4.19921875" proc.memory.rssMB‚ 2996.4296875 cpu‚25.158334608492694 disk.nvme1n1.in‚ 0.0546875! disk.sdb2.out‚270.56787109375* proc.memory.percent‚18.288755416870117 disk.rbd2.in‚0 disk.rbd0.in‚0D2Iž¦‚(:“ ÷“êÔ”ÓÛÿ gpu.0.gpu‚25.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚36.0 gpu.0.powerWatts‚119.368' gpu.0.enforcedPowerLimitWatts‚400.0) gpu.0.powerPercent‚29.841999999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0›øT‹c§‚(:Ø ÷“êÔ•¸œ€ disk.sda1.in‚0 disk.sda1.out‚0 disk.md0.out‚ 262.55859375" proc.memory.rssMB‚ 2996.4296875! disk.sdb2.out‚272.57568359375 disk.sdb.in‚0.03125 disk.rbd0.out‚ 0.63330078125 disk.rbd2.in‚0 disk.sdb2.in‚0.03125 disk.sda.in‚ 0.0390625 network.sent‚ 4.011058e+06 network.recv‚808215! disk.sda2.out‚272.57568359375 disk.sdb1.in‚0* proc.memory.percent‚18.288755416870117 cpu‚25.157678764625363 disk.md0p1.in‚ 0.0703125# disk.nvme0n1.out‚13433.84765625 disk.rbd0.in‚0 disk.nvme1n1.in‚ 0.0546875 disk.rbd2.out‚ 0.783203125% disk./.usageGB‚2711.5023880004883 disk.nvme0n1.in‚ 0.02734375 disk.sda2.in‚ 0.0390625 disk.md0.in‚ 0.0703125$ memory_percent‚99.97589588165283 disk.dm-0.in‚ 0.08203125 disk.sda.out‚272.57568359375" disk.nvme1n1.out‚ 12356.7109375 disk.sdb1.out‚0) disk./.usagePercent‚75.81474162040728 disk.dm-0.out‚25790.55859375 disk.md0p1.out‚ 262.55859375 disk.sdb.out‚272.57568359375& proc.memory.availableMB‚ 3.94921875 proc.cpu.threads‚17Å@8둨‚(:† †”êÔ ßÁÿ gpu.0.gpu‚20.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚37.0 gpu.0.powerWatts‚114.74' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.685 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0©bªUY©‚(:Î †”êÔ’ù¦€ disk.rbd0.in‚0 network.sent‚ 4.017322e+06 disk.sda.in‚ 0.0390625 disk.rbd2.in‚0 disk.sda1.in‚0 disk.rbd2.out‚ 0.783203125 disk.sdb2.in‚0.03125% disk./.usageGB‚2711.6898307800293# disk.nvme0n1.out‚13434.16015625 disk.sda2.in‚ 0.0390625 disk.rbd0.out‚ 0.63330078125 disk.md0.in‚ 0.0703125 disk.md0p1.out‚ 264.32421875 disk.dm-0.out‚26127.81640625 disk.sdb.out‚ 274.419921875 disk.sdb1.out‚0 disk.sdb2.out‚ 274.419921875 disk.dm-0.in‚ 0.08203125 cpu‚25.156804859330904 disk.md0p1.in‚ 0.0703125 disk.sda.out‚ 274.419921875 disk.nvme1n1.in‚ 0.0546875 disk.nvme1n1.out‚ 12693.65625& proc.memory.availableMB‚ 4.08203125" proc.memory.rssMB‚ 2996.4296875 disk.sda1.out‚0 network.recv‚812659 disk.nvme0n1.in‚ 0.02734375 disk.sdb.in‚0.03125$ memory_percent‚99.97508525848389 disk.sda2.out‚ 274.419921875 disk.sdb1.in‚0 disk.md0.out‚ 264.32421875* proc.memory.percent‚18.288755416870117 proc.cpu.threads‚17) disk./.usagePercent‚75.819982598974155”Cr2ª- _wandbruntime‚2414  _runtime‚2414|ˆv²Ç«Á _step‚167 & lr-AdamW/pg1‚0.0008436198164680804 & lr-AdamW/pg2‚0.0008436198164680804 trainer/global_step‚2099 % _timestamp‚1.7885127889303713e+09 _runtime‚2414.400827172ÖÖëºÌ¬Æ & lr-AdamW/pg1‚0.0008436198164680804 & lr-AdamW/pg2‚0.0008436198164680804 trainer/global_step‚2099 % _timestamp‚1.7885127889303713e+09 _runtime‚2414.400827172 _step‚167§ÎU)h2­- _wandbruntime‚2414  _runtime‚2414¥p÷Wî'®èO M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955590144 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899406848 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177046016 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2471719182336 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31684 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚95111233 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚39022795 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214657 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚172991945508864 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚170624032877056 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚42758844 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2368830041600 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411412 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚52326303 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚172992862918656 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2471490321408 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899406848 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26088 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚95085145 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚52352389 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚52352389 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚26590373 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚42758844 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26086 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚170624014874112 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚172991945508864 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚170624014874112 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177046016 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 252801024 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚39022152 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1204 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 % _timestamp‚1.7885127889315064e+09 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917409792 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚643 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0êŸF¯¯íO TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚641 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450772 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 _step‚168 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚172992862918656 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚191084003487744 trainer/global_step‚2099 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26086 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2367930634752 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1179 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2359992074281 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31929 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31684 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚172992862918656 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2471490321408 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899406848 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26088 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚95085145 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚52352389 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚52352389 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚26590373 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚42758844 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26086 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚170624014874112 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚172991945508864 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚170624014874112 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177046016 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 252801024 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚39022152 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1204 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 % _timestamp‚1.7885127889315064e+09 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917409792 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚643 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955590144 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899406848 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177046016 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2471719182336 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31684 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚95111233 QDP]6D DeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚39022795 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214657 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚172991945508864 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚170624032877056 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚42758844 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2368830041600 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411412 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚52326303 IDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955590144 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177046016 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚95155748 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚173073606030848 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚26602751 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚191173298768384 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2472662611968 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2369942858240 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚95129634 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚52377006 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 398533632 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚95155748 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12437438 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31684 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚39040816 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214657 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚42778742 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚629 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917423104 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚173073606030848 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚26602749 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2369043438080 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚39040187 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚170703663172608 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚193645961380352 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚169353456831547 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411620ï #4ü&´öM NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚42778740 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2472880973824 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26114 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚42778740 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214657 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚42778742 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚629 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917423104 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚173073606030848 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚26602749 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2369043438080 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚39040187 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚170703663172608 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚193645961380352 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚169353456831547 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411620 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26112 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26112 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1179 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚52350894 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31929 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12438065 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚627 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚169354371282527 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007233 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚52377006 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2369942858240 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955590144 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177046016 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚95155748 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚173073606030848 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚26602751 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚191173298768384 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2472662611968 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2369942858240 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚95129634 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚52377006 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 398533632 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚95155748 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12437438 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31684 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚39040816 Bv`¶Ú /loss/FBt3/detection_loss‚2.1599468663334846 epoch‚1 trainer/global_step‚2099 0loss/FRt3/assignment_loss‚3.1234701660772166 &loss/total_loss‚44.689711758487846 2loss/FBt4/assignment_loss‚0.009287333709835366 .loss/FRt1/detection_loss‚1.895824152799015 2loss/SRqqt3/assignment_loss‚1.8354579366561843 2loss/FBt3/assignment_loss‚0.060733925669858024 0loss/FBt2/assignment_loss‚0.0968934055248869 /loss/FRt3/detection_loss‚2.3111260421574116 _runtime‚2415.349631198 _step‚170 2loss/SRqqt2/assignment_loss‚0.9989004282132637 1loss/FBt1/assignment_loss‚0.07370772338860358 0loss/SRqqt1/detection_loss‚2.170887077227235 1loss/SRqqt2/detection_loss‚0.9321506574580615 1loss/SRqqt3/detection_loss‚2.2111789307943206 % _timestamp‚1.7885127898795066e+09 /loss/FRt2/detection_loss‚1.6906291427450222 0loss/SRqqt4/detection_loss‚2.616912409237453 .loss/FRt4/detection_loss‚2.770705284550786 .loss/FBt4/detection_loss‚4.401370644569397 0loss/FRt1/assignment_loss‚3.5427648524443307 0loss/FRt2/assignment_loss‚1.8104597978424608 0loss/FRt4/assignment_loss‚3.2325151674449444 2loss/SRqqt4/assignment_loss‚1.9013353769268309 2loss/SRqqt1/assignment_loss‚1.5162612893618643 /loss/FBt1/detection_loss‚2.6811962525049844 /loss/FBt2/detection_loss‚0.6459968948503956Ìk¬e·ß 2loss/SRqqt3/assignment_loss‚1.8354579366561843 2loss/FBt3/assignment_loss‚0.060733925669858024 0loss/FBt2/assignment_loss‚0.0968934055248869 /loss/FRt3/detection_loss‚2.3111260421574116 _runtime‚2415.349631198 _step‚170 2loss/SRqqt2/assignment_loss‚0.9989004282132637 1loss/FBt1/assignment_loss‚0.07370772338860358 0loss/SRqqt1/detection_loss‚2.170887077227235 1loss/SRqqt2/detection_loss‚0.9321506574580615 1loss/SRqqt3/detection_loss‚2.2111789307943206 % _timestamp‚1.7885127898795066e+09 /loss/FRt2/detection_loss‚1.6906291427450222 0loss/SRqqt4/detection_loss‚2.616912409237453 .loss/FRt4/detection_loss‚2.770705284550786 .loss/FBt4/detection_loss‚4.401370644569397 0loss/FRt1/assignment_loss‚3.5427648524443307 0loss/FRt2/assignment_loss‚1.8104597978424608 0loss/FRt4/assignment_loss‚3.2325151674449444 2loss/SRqqt4/assignment_loss‚1.9013353769268309 2loss/SRqqt1/assignment_loss‚1.5162612893618643 /loss/FBt1/detection_loss‚2.6811962525049844 /loss/FBt2/detection_loss‚0.6459968948503956 /loss/FBt3/detection_loss‚2.1599468663334846 epoch‚1 trainer/global_step‚2099 0loss/FRt3/assignment_loss‚3.1234701660772166 &loss/total_loss‚44.689711758487846 2loss/FBt4/assignment_loss‚0.009287333709835366 .loss/FRt1/detection_loss‚1.895824152799015ª f‹’¸‚(:‡ ¤”êÔË×€ gpu.0.gpu‚47.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚162.204' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚40.551 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0/Â]ñ\¹‚(:Ñ ¤”êÔ‘‚×€ disk.nvme0n1.in‚ 0.02734375 disk.sdb.out‚277.3271484375 disk.sda2.in‚ 0.0390625 disk.nvme1n1.in‚ 0.0546875 disk.sdb2.in‚0.03125 disk.sdb1.in‚0 disk.md0.in‚ 0.0703125 disk.md0.out‚ 267.1171875 disk.rbd2.in‚0 proc.cpu.threads‚17) disk./.usagePercent‚75.81143557055364 disk.dm-0.out‚26356.78515625 disk.sda.out‚277.3271484375 disk.sda1.in‚0 disk.dm-0.in‚ 0.08203125 disk.sda2.out‚277.3271484375" disk.nvme1n1.out‚ 12922.5234375$ memory_percent‚99.97761249542236* proc.memory.percent‚18.288755416870117 network.recv‚821393 disk.sdb2.out‚277.3271484375 disk.rbd2.out‚ 0.783203125 disk.md0p1.in‚ 0.0703125 disk.md0p1.out‚ 267.1171875 disk.rbd0.in‚0" proc.memory.rssMB‚ 2996.4296875# disk.nvme0n1.out‚13434.26171875 disk.sdb1.out‚0& proc.memory.availableMB‚ 3.66796875 cpu‚25.155288003812238 network.sent‚ 4.118744e+06$ disk./.usageGB‚2711.384147644043 disk.sda1.out‚0 disk.sdb.in‚0.03125 disk.sda.in‚ 0.0390625 disk.rbd0.out‚ 0.63330078125À؛擺‚(:ˆ ³”êÔ¹¥Âÿ gpu.0.gpu‚2.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚154.735' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚38.68375 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0à~R»‚(:Ç ³”êÔ®À©€ disk.md0.in‚ 0.0703125# memory_percent‚99.9779462814331) disk./.usagePercent‚75.81143354400352 disk.sda1.in‚0# disk.nvme1n1.out‚13066.99609375 disk.rbd2.out‚ 0.783203125" proc.memory.rssMB‚ 2996.4296875 disk.md0p1.out‚ 269.09375" disk.nvme0n1.out‚ 13434.2734375 disk.rbd2.in‚0 network.recv‚826465 disk.sda2.out‚ 279.365234375 disk.dm-0.in‚ 0.08203125 disk.nvme0n1.in‚ 0.02734375 disk.sdb.in‚0.03125 disk.rbd0.in‚0 disk.rbd0.out‚ 0.63330078125& proc.memory.availableMB‚ 3.61328125 cpu‚25.15435299227312 network.sent‚ 4.215582e+06 disk.sdb.out‚ 279.365234375* proc.memory.percent‚18.288755416870117 disk.sda1.out‚0 disk.dm-0.out‚26501.26953125 disk.sda.out‚ 279.365234375 disk.sda2.in‚ 0.0390625 disk.nvme1n1.in‚ 0.0546875 disk.sdb1.out‚0 disk.md0.out‚ 269.09375 proc.cpu.threads‚17$ disk./.usageGB‚2711.384075164795 disk.sdb2.out‚ 279.365234375 disk.sda.in‚ 0.0390625 disk.sdb1.in‚0 disk.sdb2.in‚0.03125 disk.md0p1.in‚ 0.0703125¦º/ ”¼‚(:‰ ”êԫݯÿ gpu.0.gpu‚42.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚168.051' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.01275 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0L†‰\½‚(:Ñ ”êÔ¡Éš€) disk./.usagePercent‚75.80901352319022 disk.rbd2.out‚ 0.783203125 disk.dm-0.out‚ 26502.453125 disk.nvme0n1.in‚ 0.02734375 proc.cpu.threads‚17 disk.sdb1.in‚0! disk.sdb2.out‚280.84521484375 disk.dm-0.in‚ 0.08203125 disk.md0p1.out‚ 270.5078125# disk.nvme1n1.out‚13068.15234375 network.sent‚ 4.223945e+06 disk.sdb1.out‚0# disk.nvme0n1.out‚13434.30078125 disk.sda2.in‚ 0.0390625! disk.sda2.out‚280.84521484375 disk.nvme1n1.in‚ 0.0546875# memory_percent‚99.9779224395752" proc.memory.rssMB‚ 2996.4296875$ disk./.usageGB‚2711.297523498535 disk.sda1.out‚0 disk.md0p1.in‚ 0.0703125* proc.memory.percent‚18.288755416870117 disk.rbd0.out‚ 0.63330078125 disk.sda1.in‚0 disk.sdb.out‚280.84521484375 disk.sda.out‚280.84521484375 cpu‚25.15362500035511 disk.sda.in‚ 0.0390625 disk.rbd0.in‚0 network.recv‚830448 disk.md0.in‚ 0.0703125 disk.md0.out‚ 270.5078125 disk.sdb2.in‚0.03125 disk.rbd2.in‚0 disk.sdb.in‚0.03125% proc.memory.availableMB‚ 3.6171875øùi¸2¾- _wandbruntime‚2466  _runtime‚2466â/_\Æ¿À & lr-AdamW/pg1‚0.0008420032570655903 & lr-AdamW/pg2‚0.0008420032570655903 trainer/global_step‚2149 $ _timestamp‚1.788512840853301e+09 _runtime‚2466.323871378 _step‚171Å‹‹sÀÅ & lr-AdamW/pg1‚0.0008420032570655903 & lr-AdamW/pg2‚0.0008420032570655903 trainer/global_step‚214C«AX9 $ _timestamp‚1.788512840853301e+09 _runtime‚2466.323871378 _step‚171«ìL9~2Á- _wandbruntime‚2466  _runtime‚2466«yx-î'ÂèO VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007233 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚174605529651712 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 398533632 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚170814295158992 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899406848 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚195549940584448 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚174605529651712 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2529602152448 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚39929040 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚97312947 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚53559205 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31684 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2424484621824 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚43753742 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚27209173 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚177029114866688 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚53585291 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚198079780857856 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917409792 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚53559205 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2423585214976 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450772 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26086 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2424484621824 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12719865 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚177029114866688 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2423585214976 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899406848 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12719261 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2415438731389 _step‚172 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚177030032276480 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚174605547654656 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2529816333312 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚173228819439609 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚177030032276480 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚39928434 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1204 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚604 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31929 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚43753744 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26086 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31929 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚43753742 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚606 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411412 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 _runtime‚2466.325173693 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2414541319977 % _timestamp‚1.7885128408545506e+09 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚53585291 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚174605547654656 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚195549964524544 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 214180864 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214657 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚97339035 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31684 trainer/global_step‚2149 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚43753744 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177046016 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955590144 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26088 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚27209175 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177046016 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26088 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚27209175 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177046016 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31684 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2424484621824 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚43753742 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚27209173 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚177029114866688 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚53585291 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚198079780857856 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917409792 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚53559205 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2423585214976 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450772 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26086 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2424484621824 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12719865 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚177029114866688 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2423585214976 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899406848 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12719261 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2415438731389 _step‚172 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚177030032276480 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚174605547654656 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2529816333312 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚173228819439609 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚177030032276480 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚39928434 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1204 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚604¬ Þò³2Ä- _wandbruntime‚2467  _runtime‚2467n.*õ÷&ÅñM ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚174685159947264 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚27221549 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚39947096 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 _runtime‚2467.301835629 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 248593408 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚177110775388672 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917423104 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚174685177950208 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚97383550 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2416547392915 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚27221551 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚53583796 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚170892275253120 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚174685177950208 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2425597438464 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899420160 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12724946 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955590144 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚43773642 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚43773640 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1179 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26112 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚599 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31929 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2424698018304 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214657 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2415649981295 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚39946495 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007233 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚53609908 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26114 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚43773640 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚97357436 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450980 trainer/global_step‚2149 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31929 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 224653312 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411620 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚174685159947264 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚198170021249536 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚97357436 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2424698018304 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚601 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177046016 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚195639283745280 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 $ _timestamp‚1.788512841830264e+09 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚177110775388672 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚53609908 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712°rû;ÆöM F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26112 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚599 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31929 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2424698018304 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214657 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2415649981295 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚39946495 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007233 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚53609908 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26114 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚43773640 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚97357436 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450980 trainer/global_step‚2149 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31929 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 224653312 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411620 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚174685159947264 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚198170021249536 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚97357436 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2424698018304 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚601 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177046016 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚195639283745280 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 $ _timestamp‚1.788512841830264e+09 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚177110775388672 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚53609908 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚174685159947264 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚27221549 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚39947096 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 _runtime‚2467.301835629 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 248593408 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚177110775388672 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917423104 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚174685177950208 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚97383550 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2416547392915 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚27221551 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚53583796 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚170892275253120 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚174685177950208 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2425597438464 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899420160 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12724946 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955590144 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚43773642 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚43773640 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1179­Ô©kù2Ç- _wandbruntime‚2467  _runtime‚2467¨ŸÅXÈÒ 0loss/FRt1/assignment_loss‚3.6800019308632494 0loss/FBt3/assignment_loss‚0.4513442143132644 1loss/SRqqt2/detection_loss‚1.0700954028964043 _runtime‚2467.302803584 0loss/SRqqt1/assignment_loss‚1.41220552350084 1loss/SRqqt4/assignment_loss‚2.093221327340281 0loss/SRqqt1/detection_loss‚2.085022402582345 /loss/FBt4/detection_loss‚2.7767630368471146 0loss/FRt2/assignment_loss‚1.1299559116363525 .loss/FRt1/detection_loss‚2.104186583383411 1loss/SRqqt3/detection_loss‚1.9364729277465655 epoch‚1 trainer/global_step‚2149 % _timestamp‚1.7885128418326745e+09 /loss/FRt4/assignment_loss‚3.623151333101334 2loss/SRqqt2/assignment_loss‚0.7837060839434464 2loss/SRqqt3/assignment_loss‚1.6698513724233792 /loss/FRt2/detection_loss‚1.4114192122085527 /loss/FRt3/detection_loss‚2.4165945490201315 _step‚174 0loss/FBt1/assignment_loss‚0.8908365902607329 1loss/SRqqt4/detection_loss‚2.3621568051544397 %loss/total_loss‚44.94462045759553 0loss/FRt3/assignment_loss‚3.1177946779463026 /loss/FBt3/detection_loss‚2.7745117034230913 0loss/FBt2/assignment_loss‚0.1457218945840004 0loss/FBt4/assignment_loss‚0.2588052271530614 .loss/FRt4/detection_loss‚2.794368318973049 /loss/FBt1/detection_loss‚3.1965919584035873 /loss/FBt2/detection_loss‚0.7598414698905953pËœ­]É× 0loss/FRt1/assignment_loss‚3.6800019308632494 0loss/FBt3/assignment_loss‚0.4513442143132644 1loss/SRqqt2/detection_loss‚1.0700954028964043 _runtime‚2467.302803584 0loss/SRqqt1/assignment_loss‚1.41220552350084 1loss/SRqqt4/assignment_loss‚2.093221327340281 0loss/SRqqt1/detection_loss‚2.085022402582345 /loss/FBt4/detection_loss‚2.7767630368471146 0loss/FRt2/assignment_loss‚1.1299559116363525 .loss/FRt1/detection_loss‚2.104186583383411 1loss/SRqqt3/detection_loss‚1.9364729277465655 epoch‚1 trainer/global_step‚2149 % _timestamp‚1.7885128418326745e+09 /loss/FRt4/assignment_loss‚3.623151333101334 2loss/SRqqt2/assignment_loss‚0.7837060839434464 2loss/SRqqt3/assignment_loss‚1.6698513724233792 /loss/FRt2/detection_loss‚1.4114192122085527 /loss/FRt3/detection_loss‚2.4165945490201315 _step‚174 0loss/FBt1/assignment_loss‚0.8908365902607329 1loss/SRqqt4/detection_loss‚2.3621568051544397 %loss/total_loss‚44.94462045759553 0loss/FRt3/assignment_loss‚3.1177946779463026 /loss/FBt3/detection_loss‚2.7745117034230913 0loss/FBt2/assignment_loss‚0.1457218945840004 0loss/FBt4/assignment_loss‚0.2588052271530614 .loss/FRt4/detection_loss‚2.794368318973049 /loss/FBt1/detection_loss‚3.1965919584035873 /loss/FBt2/detection_loss‚0.7598414698905953®‡Ìb“Ê‚(:ˆ Ñ”êÔ‚©±€ gpu.0.gpu‚50.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚167.434' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚41.8585 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚07ÔÄXË‚(:Í Ñ”êÔÏ­ó disk.dm-0.in‚ 0.08203125 disk.md0p1.out‚ 272.01953125 disk.sda2.in‚ 0.0390625 disk.sda1.out‚0 disk.md0p1.in‚ 0.0703125) disk./.usagePercent‚75.81289191345923 disk.nvme1n1.in‚ 0.0546875 disk.sda.in‚ 0.0390625" proc.memory.rssMB‚ 2996.4296875$ disk./.usageGB‚2711.436233520508 disk.sdb.in‚0.03125 disk.sda2.out‚ 282.408203125 disk.rbd0.in‚0& proc.memory.availableMB‚ 3.78515625 disk.rbd2.out‚ 0.783203125 disk.sda1.in‚0 disk.dm-0.out‚26718.01953125 disk.nvme0n1.in‚ 0.02734375 proc.cpu.threads‚17 network.sent‚ 4.239139e+06 disk.sdb1.in‚0 disk.md0.in‚ 0.0703125 disk.sdb.out‚ 282.408203125$ memory_percent‚99.97689723968506 disk.nvme1n1.out‚ 13212.71875 disk.sdb1.out‚0 disk.md0.out‚ 272.01953125 disk.rbd2.in‚0 disk.sdb2.out‚ 282.408203125 disk.rbd0.out‚ 0.63330078125 cpu‚25.152968353831373# disk.nvme0n1.out‚13505.30078125 disk.sda.out‚ 282.408203125* proc.memory.percent‚18.288755416870117 network.recv‚833650 disk.sdb2.in‚0.03125Î]{C”Ì‚(:‰ à”êÔ…‰þ gpu.0.gpu‚10.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚170.047' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.51175 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ç|€DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚177184185253889 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955590144 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2469987977085 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚44748642 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177046016 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450772 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2480139202048 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚202604183336448 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚174713299865392 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚54818193 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 398533632 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2479239795200 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 _step‚176 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚181066284224512 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26088 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚44748642 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚27827973 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31684 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31929 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚99566837 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚181067201634304 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13008465 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2588045691392 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 trainer/global_step‚2199 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26086 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2480139202048 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899406848 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚44748644 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚174713282826032 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26086 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚200015925561344 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚178587044429312 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2479239795200 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚54792107 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚593 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1179 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚200015901621248 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚27827975 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177046016 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214657 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚178587062432256 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚177183270803117 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚54818193 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚44748644 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917409792 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚40837033 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚595 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1204 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚99540749WEÓBFÕíO BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚181066284224512 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26088 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚44748642 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚27827973 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31684 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31929 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚99566837 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚181067201634304 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13008465 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2588045691392 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 trainer/global_step‚2199 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26086 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2480139202048 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899406848 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚44748644 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚174713282826032 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26086 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚200015925561344 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚178587044429312 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2479239795200 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚54792107 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚593 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1179 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚200015901621248 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚27827975 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177046016 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214657 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚178587062432256 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚177183270803117 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚54818193 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚44748644 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917409792 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚40837033 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚595 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1204 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚99540749 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917409792 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚54792107 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31684 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26088 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚181067201634304 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 236023808 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007233 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2470885388497 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411412 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955590144 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899406848 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚181066284224512 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚178587044429312 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 212083712 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 % _timestamp‚1.7885128937721832e+09 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚178587062432256 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13009058 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_b¨Ãuä­ ytes.all.allocated‚177184185253889 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955590144 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2469987977085 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚44748642 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177046016 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450772 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2480139202048 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚202604183336448 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚174713299865392 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚54818193 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 398533632 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2479239795200 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 _step‚176 IDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚99611352 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚178666674724864 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411620 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177046016 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚621 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚27840349 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2480352598528 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚44768542 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007233 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917423104 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899420160 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚177263274009543 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214657 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚54842810 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚2199 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26114 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚181147944746496 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955590144 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _step‚177 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26112 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚27840351 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚44768540 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1204 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚177262359558563 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917423104 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚54816698 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 398533632 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚181147027323392 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚99585238 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 233913344 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 209973248 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚178666692727808 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31929 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚174791262920160 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2471994050023 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚181147944746496 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177046016 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚200105220841984 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚44768542 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚40854617 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2589449788928 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚174791279959520 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712òÉcü&ÚöM NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚54816698 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 398533632 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚181147027323392 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚99585238 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 233913344 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 209973248 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚178666692727808 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31929 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚174791262920160 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2471994050023 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚181147944746496 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177046016 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚200105220841984 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚44768542 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚40854617 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2589449788928 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚174791279959520 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899420160 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚54842810 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2480352598528 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31684 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1179 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚54816698 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955590144 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚178666692727808 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚99585238 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13014889 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚202694694571008 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚40855240 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2589239815680 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚99611352 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚99611352 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚178666674724864 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411620 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177046016 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚621 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚27840349 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2480352598528 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚44768542 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007233 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917423104 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899420160 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚177263274009543 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214657 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚54842810 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚2199 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26114 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚181147944746496 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955590144 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _step‚177 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26112 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚27840351 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚44768540 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1204 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚177262359558563 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917423104±@RŸ2Û- _wandbruntime‚2520  _runtime‚25200£8\ÜÖ 0loss/FBt1/assignment_loss‚0.1895140716806054 1loss/SRqqt1/detection_loss‚1.7599191259254108 /loss/FRt3/assignment_loss‚2.816458872720307 1loss/SRqqt2/detection_loss‚0.9177433119519897 %loss/total_loss‚43.11162422661042 _step‚178 2loss/SRqqt3/assignment_loss‚1.6272945324114214 /loss/FRt1/detection_loss‚2.0729470478163874 2loss/SRqqt1/assignment_loss‚1.6824841761453586 1loss/FBt4/assignment_loss‚0.18394837511533524 .loss/FBt1/detection_loss‚3.139506086707115 /loss/FBt2/detection_loss‚0.6829323462929718 2loss/SRqqt2/assignment_loss‚1.0125763525349507 /loss/FRt3/detection_loss‚2.3023252989731584 0loss/SRqqt4/detection_loss‚2.459059154821767 0loss/FRt2/assignment_loss‚1.4767582919083388 2loss/SRqqt4/assignment_loss‚2.1528421797686152 1loss/FBt3/assignment_loss‚0.29873141758089333 1loss/SRqqt3/detection_loss‚1.7098000386419396 epoch‚1 trainer/global_step‚2199 _runtime‚2520.254433366 /loss/FRt4/assignment_loss‚2.845611148410373 1loss/FBt2/assignment_loss‚0.17698186198430788 /loss/FRt2/detection_loss‚1.6239176673047684 /loss/FRt4/detection_loss‚2.8750461753871708 /loss/FBt4/detection_loss‚2.6474124030633406 /loss/FRt1/assignment_loss‚3.252138532532586 .loss/FBt3/detection_loss‚3.205675756931305 % _timestamp‚1.7885128947842977e+092LÕ³aÝÛ /loss/FRt3/assignment_loss‚2.816458872720307 1loss/SRqqt2/detection_loss‚0.9177433119519897 %loss/total_loss‚43.11162422661042 _step‚178 2loss/SRqqt3/assignment_loss‚1.6272945324114214 /loss/FRt1/detection_loss‚2.0729470478163874 2loss/SRqqt1/assignment_loss‚1.6824841761453586 1loss/FBt4/assignment_loss‚0.18394837511533524 .loss/FBt1/detection_loss‚3.139506086707115 /loss/FBt2/detection_loss‚0.6829323462929718 2loss/SRqqt2/assignment_loss‚1.0125763525349507 /loss/FRt3/detection_loss‚2.3023252989731584 0loss/SRqqt4/detection_loss‚2.459059154821767 0loss/FRt2/assignment_loss‚1.4767582919083388 2loss/SRqqt4/assignment_loss‚2.1528421797686152 1loss/FBt3/assignment_loss‚0.29873141758089333 1loss/SRqqt3/detection_loss‚1.7098000386419396 epoch‚1 trainer/global_step‚2199 _runtime‚2520.254433366 /loss/FRt4/assignment_loss‚2.845611148410373 1loss/FBt2/assignment_loss‚0.17698186198430788 /loss/FRt2/detection_loss‚1.6239176673047684 /loss/FRt4/detection_loss‚2.8750461753871708 /loss/FBt4/detection_loss‚2.6474124030633406 /loss/FRt1/assignment_loss‚3.252138532532586 .loss/FBt3/detection_loss‚3.205675756931305 % _timestamp‚1.7885128947842977e+09 0loss/FBt1/assignment_loss‚0.1895140716806054 1loss/SRqqt1/detection_loss‚1.7599191259254108²á“Þ‚(:ˆ •êÔ¼ì€ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚36.0 gpu.0.powerWatts‚114.131' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.53275 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0âÄ6BMß‚(: •êÔÒÇׂ disk.sda2.in‚ 0.0390625 disk.nvme1n1.in‚ 1.42578125 disk.sdb1.in‚0 disk.dm-0.out‚ 27324.453125 disk.md0p1.out‚278.6875" disk.nvme0n1.out‚ 13830.5546875* proc.memory.percent‚18.288779258728027 disk.md0.out‚278.6875 disk.rbd2.out‚ 0.783203125 disk.sdb2.in‚0.03125 disk.sda1.in‚0 disk.dm-0.in‚ 2.86328125 cpu‚25.15005069697536 network.sent‚ 4.441739e+06 disk.sdb1.out‚0 disk.rbd2.in‚0 network.recv‚852470 disk.sdb.out‚ 289.322265625 disk.sda.in‚ 0.0390625 disk.sdb2.out‚ 289.322265625$ memory_percent‚99.97785091400146& proc.memory.availableMB‚ 3.62890625) disk./.usagePercent‚75.81191852944309 disk.rbd0.in‚0 disk.md0p1.in‚ 0.0703125 disk.nvme0n1.in‚1.4375% disk./.usageGB‚2711.4014205932617 disk.sda2.out‚ 289.322265625 disk.md0.in‚ 0.0703125 disk.sda1.out‚0# proc.memory.rssMB‚ 2996.43359375 proc.cpu.threads‚17 disk.rbd0.out‚ 0.63330078125 disk.sdb.in‚0.03125 disk.sda.out‚ 289.322265625" disk.nvme1n1.out‚ 13493.8984375ª î9‘à‚(:† œ•êÔ•ñáÿ gpu.0.gpu‚56.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚172.22' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.055 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0àÞú‰]á‚(:Ò œ•êÔ¯‡€€" disk.nvme1n1.out‚ 13494.0703125 disk.sdb1.in‚0 disk.md0.in‚ 0.0703125 disk.md0.out‚ 280.07421875 disk.sdb2.in‚0.03125 disk.dm-0.out‚ 27554.3515625 disk.md0p1.out‚ 280.07421875$ memory_percent‚99.97942447662354 disk.sda2.in‚ 0.0390625 disk.sda1.out‚0& proc.memory.availableMB‚ 3.37109375% disk./.usageGB‚2711.4401779174805 disk.sdb.out‚290.77294921875 disk.rbd0.out‚ 0.63330078125 disk.rbd2.in‚0 disk.rbd2.out‚ 0.783203125 disk.sda1.in‚0 disk.dm-0.in‚ 2.86328125# proc.memory.rssMB‚ 2996.43359375 disk.nvme0n1.in‚1.4375! disk.sda2.out‚290.77294921875 disk.sdb1.out‚0 disk.md0p1.in‚ 0.0703125* proc.memory.percent‚18.288779258728027) disk./.usagePercent‚75.81300220044945 disk.nvme0n1.out‚ 14060.28125 disk.sda.in‚ 0.0390625! disk.sdb2.out‚290.77294921875 cpu‚25.149328498403854 network.sent‚ 4.540788e+06 disk.sda.out‚290.77294921875 disk.rbd0.in‚0 disk.nvme1n1.in‚ 1.42578125 network.recv‚857697 disk.sdb.in‚0.03125 proc.cpu.threads‚17€ ž“â‚(:ˆ «•êÔÝÉ÷ÿ gpu.0.gpu‚2.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚166.207' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚41.55175 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0x«{=\ã‚(:Ñ «•êÔÁÞû proc.cpu.threads‚17) disk./.usagePercent‚75.81649554619749 disk.nvme1n1.in‚ 1.42578125 disk.md0p1.in‚ 0.0703125 disk.rbd0.out‚ 0.63330078125 disk.sdb1.out‚0 disk.md0.out‚ 282.1171875 disk.sdb2.in‚0.03125 disk.sda.in‚ 0.0390625$ disk./.usageGB‚2711.565116882324 disk.sdb.in‚0.03125$ memory_percent‚99.97642040252686# disk.nvme1n1.out‚13494.84765625 disk.rbd2.in‚0 disk.nvme0n1.in‚1.4375# disk.nvme0n1.out‚14247.16015625& proc.memory.availableMB‚ 3.86328125 disk.sda1.out‚0 disk.dm-0.out‚ 27742.0078125 disk.md0p1.out‚ 282.1171875 disk.sda.out‚292.88427734375! disk.sda2.out‚292.88427734375 disk.rbd0.in‚0 disk.sdb1.in‚0 disk.sdb.out‚292.88427734375 disk.sda2.in‚ 0.0390625* proc.memory.percent‚18.288779258728027 cpu‚25.14852018657167 disk.rbd2.out‚ 0.783203125! disk.sdb2.out‚292.88427734375 disk.sda1.in‚0 network.sent‚ 4.54668e+06 network.recv‚861675 disk.md0.in‚ 0.0703125 disk.dm-0.in‚ 2.86328125# proc.memory.rssMB‚ 2996.43359375ã×·Û2ä- _wandbruntime‚2576  _runtime‚2576µÁx¯ÇåÁ & lr-AdamW/pg1‚0.0008385121917214872 & lr-AdamW/pg2‚0.0008385121917214872 trainer/global_step‚2249 % _timestamp‚1.7885129510363483e+09 _runtime‚2576.506998763 _step‚179Er\æÆ & lr-AdamW/pg2‚0.00T©z8¯08385121917214872 trainer/global_step‚2249 % _timestamp‚1.7885129510363483e+09 _runtime‚2576.506998763 _step‚179 & lr-AdamW/pg1‚0.0008385121917214872³— .‘2ç- _wandbruntime‚2576  _runtime‚2576»P]ð'èêO TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚28446773 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚181138636617397 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31684 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177046016 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917409792 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚204481886598144 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚181137722166625 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007233 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2535793782272 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899406848 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 trainer/global_step‚2249 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31929 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚185104370992128 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚41745089 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚41744454 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1180 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214657 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2525434634193 $ _timestamp‚1.788512951037689e+09 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚56025009 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚185103453582336 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955590144 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26088 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2646466015232 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚28446775 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13298314 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955590144 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚633 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚182568577209856 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 101794639 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚635 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1205 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚45743542 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚204481862658048 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177046016 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚207128099812352 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚182568577209856 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚45743544 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26086 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 398533632 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 252801024 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚178612287532432 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917409792 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2646237154304 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31929 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚45743542 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31684 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26086 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚178612304571792 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚207128352613376 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411412 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚182568559206912 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚56025009ãaÌiõ'éïO TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2535793782272 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899406848 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 trainer/global_step‚2249 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31929 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚185104370992128 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚41745089 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚41744454 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1180 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214657 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2525434634193 $ _timestamp‚1.788512951037689e+09 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚56025009 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚185103453582336 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955590144 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26088 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2646466015232 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚28446775 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13298314 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955590144 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚633 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚182568577209856 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 101794639 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚635 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1205 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚45743542 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚204481862658048 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177046016 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚207128099812352 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚182568577209856 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚45743544 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26086 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 398533632 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 252801024 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚178612287532432 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917409792 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2646237154304 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31929 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚45743542 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31684 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26086 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚178612304571792 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚207128352613376 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411412 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚182568559206912 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚56025009 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚28446773 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚181138636617397 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31684 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177046016 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917409792 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚204481886598144 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚181137722166625 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007233´Å_ý2ê- _wandbruntime‚2577  _runtime‚2577ÃAA†ü&ëöM REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚45763442 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31684 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚178690284665920 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚2249 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2536906598912 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚185184196681216 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚646 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 248593408 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚204571181878784 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2647620468736 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 % _timestamp‚1.7885129520023808e+09 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955590144 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2536906598912 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1205 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚28459151 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26112 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177046016 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚207218826287616 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 398533632 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007233 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚41763318 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 224653312 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31929 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899420160 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚185185114104320 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚56075712 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚13303523 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411620 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚182648207505408 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26112 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚181217725373051 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚181216810922071 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917423104 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955590144 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 101839154 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917423104 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31929 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚207218577694208 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13304167 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214657 _step‚181 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚45763440 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450980 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚182648189502464 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899420160 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚56049600 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚178690267626560 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 101813040 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚182648189502464 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177046016 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚28459149 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚204571205818880 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26114 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚56075712 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1180 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚45763442 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2536007178752 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2526543295511 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚56049600 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚644 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚41762672 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2536007178752 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214657 _step‚181 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚45763440 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450980 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.fÉÂøÞ&reed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚182648189502464 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899420160 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚56049600 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚178690267626560 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 101813040 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚182648189502464 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177046016 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚28459149 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚204571205818880 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26114 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚56075712 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1180 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚45763442 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2536007178752 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2526543295511 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚56049600 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚644 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚41762672 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2536007178752 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚2249 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2536906598912 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚185184196681216 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚646 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 248593408 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚204571181878784 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2647620468736 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 % _timestamp‚1.7885129520023808e+09 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955590144 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2536906598912 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1205 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚28459151 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26112 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177046016 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚207218826287616 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 398533632 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007233 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚41763318 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 224653312 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31929 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899420160 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚185185114104320 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚56075712 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚13303523 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411620 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚182648207505408 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26112 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚181217725373051 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚181216810922071 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917423104 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955590144 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 101839154 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917423104 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31929µ>žhç2í- _wandbruntime‚2577  _runtime‚2577ÀPÃ[îÕ % _timestamp‚1.7885129520045874e+09 _runtime‚2577.474730838 _step‚182 1loss/FBt2/assignment_loss‚0.10969150647908843 1loss/SRqqt3/detection_loss‚1.7027958042919635 1loss/SRqqt4/detection_loss‚2.6892263081765946 .loss/FBt3/detection_loss‚2.711937926709652 0loss/FRt3/assignment_loss‚3.1271062081768397 .loss/FRt1/detection_loss‚2.419303221361978 .loss/FBt4/detection_loss‚4.896531899770101 /loss/FRt2/assignment_loss‚1.579695164011075 2loss/SRqqt1/assignment_loss‚1.5379013144834475 1loss/SRqqt1/detection_loss‚1.8264133247462186 1loss/SRqqt2/detection_loss‚1.0050931695800323 0loss/FRt4/assignment_loss‚4.1495713868311475 2loss/SRqqt4/assignment_loss‚2.3884483700798405 2loss/SRqqt3/assignment_loss‚1.4846661505599816 epoch‚1 0loss/FRt1/assignment_loss‚2.9707970193454196 1loss/FBt1/assignment_loss‚0.18044818689107464 /loss/FRt2/detection_loss‚1.6143781531315584 /loss/FRt3/detection_loss‚2.4976086105619157 0loss/FBt3/assignment_loss‚0.6625297794813605 /loss/FBt1/detection_loss‚2.5878984133402505 %loss/total_loss‚47.38439509855128 trainer/global_step‚2249 2loss/SRqqt2/assignment_loss‚0.9085725245380459 /loss/FBt4/assignment_loss‚0.471842048689723 .loss/FRt4/detection_loss‚2.998709738254547 /loss/FBt2/detection_loss‚0.8632288690594335cM‚`ïÚ 0loss/FBt3/assignment_loss‚0.6625297794813605 /loss/FBt1/detection_loss‚2.5878984133402505 %loss/total_loss‚47.38439509855128 trainer/global_step‚2249 2loss/SRqqt2/assignment_loss‚0.9085725245380459 /loss/FBt4/assignment_loss‚0.471842048689723 .loss/FRt4/detection_loss‚2.998709738254547 /loss/FBt2/detection_loss‚0.8632288690594335 % _timestamp‚1.7885129520045874e+09 _runtime‚2577.474730838 _step‚182 1loss/FBt2/assignment_loss‚0.10969150647908843 1loss/SRqqt3/detection_loss‚1.7027958042919635 1loss/SRqqt4/detection_loss‚2.6892263081765946 .loss/FBt3/detection_loss‚2.711937926709652 0loss/FRt3/assignment_loss‚3.1271062081768397 .loss/FRt1/detection_loss‚2.419303221361978 .loss/FBt4/detection_loss‚4.896531899770101 /loss/FRt2/assignment_loss‚1.579695164011075 2loss/SRqqt1/assignment_loss‚1.5379013144834475 1loss/SRqqt1/detection_loss‚1.8264133247462186 1loss/SRqqt2/detection_loss‚1.0050931695800323 0loss/FRt4/assignment_loss‚4.1495713868311475 2loss/SRqqt4/assignment_loss‚2.3884483700798405 2loss/SRqqt3/assignment_loss‚1.4846661505599816 epoch‚1 0loss/FRt1/assignment_loss‚2.9707970193454196 1loss/FBt1/assignment_loss‚0.18044818689107464 /loss/FRt2/detection_loss‚1.6143781531315584 /loss/FRt3/detection_loss‚2.4976086105619157¶ @¦“ð‚(:ˆ º•êÔú›¦ÿ gpu.0.gpu‚1.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚36.0 gpu.0.powerWatts‚114.131' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.53275 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0;¦S]_ñ‚(:Ô º•êÔ‹É…€ network.recv‚866036( disk./.usagePercent‚75.8200049976859" disk.nvme0n1.out‚ 14247.1640625# disk.nvme1n1.out‚13494.98828125 disk.md0.in‚ 0.0703125 disk.rbd2.in‚0 disk.md0p1.out‚ 283.49609375 disk.sdb.out‚294.32861328125$ memory_percent‚99.98509883880615 cpu‚25.147800008476967 network.sent‚ 4.550755e+06! disk.sda2.out‚294.32861328125 disk.rbd2.out‚ 0.783203125 disk.dm-0.out‚27742.15234375 disk.sda.in‚ 0.0390625 disk.nvme1n1.in‚ 1.42578125! disk.sdb2.out‚294.32861328125* proc.memory.percent‚18.288779258728027 disk.rbd0.out‚ 0.69580078125 disk.md0p1.in‚ 0.0703125 disk.sda.out‚294.32861328125 disk.rbd0.in‚0 disk.md0.out‚ 283.49609375 disk.sda1.out‚0 disk.sdb1.in‚0 disk.sdb2.in‚0.03125 disk.dm-0.in‚ 2.86328125& proc.memory.availableMB‚ 2.44140625 proc.cpu.threads‚17$ disk./.usageGB‚2711.690631866455 disk.sda2.in‚ 0.0390625 disk.sdb1.out‚0 disk.sda1.in‚0 disk.nvme0n1.in‚1.4375 disk.sdb.in‚0.03125# proc.memory.rssMB‚ 2996.43359375;tà”ò‚(:‰ É•êÔ«Žëý gpu.0.gpu‚57.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚166.825' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚41.70625 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Iy¸Mó‚(: ɕêÔçñÿ) disk./.usagePercent‚75.82698070314719$ disk./.usageGB‚2711.940116882324 disk.md0.in‚ 0.0703125 disk.sdb2.out‚ 296.00390625 disk.sda1.out‚0 disk.md0p1.out‚ 285.109375# disk.nvme0n1.out‚14623.37109375 disk.sda2.out‚ 296.00390625 network.recv‚870821 disk.sda1.in‚0$ memory_percent‚99.98064041137695 disk.nvme1n1.in‚ 1.42578125 disk.sda.out‚ 296.00390625 disk.rbd0.in‚0* proc.memory.percent‚18.288779258728027# disk.nvme1n1.out‚13495.23046875 disk.rbd2.out‚ 0.783203125 disk.md0p1.in‚ 0.0703125 disk.sdb.out‚ 296.00390625 proc.cpu.threads‚17 network.sent‚ 4.65974e+06 disk.sdb1.out‚0 disk.rbd2.in‚0$ proc.memory.availableMB‚3.171875 disk.sdb1.in‚0 disk.nvme0n1.in‚1.4375 disk.sda.in‚ 0.0390625 disk.sda2.in‚ 0.0390625# proc.memory.rssMB‚ 2996.43359375 cpu‚25.147067588086426 disk.md0.out‚ 285.109375 disk.sdb2.in‚0.03125 disk.dm-0.in‚ 2.86328125 disk.sdb.in‚0.03125 disk.dm-0.out‚ 28118.6015625 disk.rbd0.out‚ 0.695800781255øç”ô‚(:‰ Ø•êÔû¿Ãÿ gpu.0.gpu‚41.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚37.0 gpu.0.powerWatts‚152.899' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚38.22475 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0nÀBxYõ‚(:Î Ø•êÔÒïõÿ disk.rbd2.out‚ 0.783203125 disk.sdb1.out‚0 disk.sdb2.in‚0.03125 disk.sda2.in‚ 0.0390625 disk.rbd0.in‚0 network.sent‚ 4.665644e+06$ disk./.usageGB‚2711.940116882324 disk.nvme0n1.in‚1.4375& proc.memory.availableMB‚ 3.16796875* proc.memory.percent‚18.288779258728027 cpu‚25.14640470495627 disk.md0.in‚ 0.0703125 network.recv‚875181 disk.dm-0.in‚ 2.86328125 disk.sdb.out‚298.07568359375 disk.nvme1n1.in‚ 1.42578125 disk.sdb1.in‚0) disk./.usagePercent‚75.82698070314719! disk.sdb2.out‚298.07568359375 disk.sda1.in‚0 disk.dm-0.out‚ 28320.7421875 disk.md0p1.in‚ 0.0703125 disk.md0p1.out‚ 287.1015625 disk.rbd2.in‚0$ memory_percent‚99.98066425323486 disk.rbd0.out‚ 0.7197265625 disk.md0.out‚ 287.1015625" disk.nvme0n1.out‚ 14825.3515625 disk.sdb.in‚0.03125 disk.sda.in‚ 0.0390625 disk.sda.out‚298.07568359375 proc.cpu.threads‚17! disk.nvme1n1.out‚ 13495.390625# proc.memory.rssMB‚ 2996.43359375 disk.sda1.out‚0! disk.sda2.out‚298.07568359375bVšÎ“ö‚(:ˆ ç•êÔû¿æý gpu.0.gpu‚42.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚39.0 gpu.0.powerWatts‚181.414' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚45.3535 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ƒgd´X÷‚(:Í ç•êÔöƒûþ disk.rbd0.out‚ 0.7197265625 disk.nvme1n1.in‚ 1.42578125 disk.md0p1.in‚ 0.0703125# proc.memory.rssMB‚ 2996.43359375 disk.sda2.in‚ 0.0390625 disk.md0.in‚ 0.0703125 cpu‚25.145689277015432 proc.cpu.threads‚17 disk.sda.in‚ 0.0390625 disk.md0.out‚ 288.51171875# disk.nvme0n1.out‚15015.10546875$ proc.memory.availableMB‚2.921875" disk.nvme1n1.out‚ 13495.6953125 disk.sdb1.in‚0 disk.sdb2.in‚0.03125# memory_percent‚99.9821662902832 disk.rbd2.out‚ 0.783203125 disk.sdb2.out‚299.5458984375 disk.dm-0.out‚28510.80078125 disk.sda2.out‚299.5458984375 disk.nvme0n1.in‚1.4375* proc.memory.percent‚18.288779258728027 disk.rbd2.in‚0 disk.sda1.out‚0 disk.sdb.in‚0.03125 disk.md0p1.out‚ 288.51171875 network.recv‚879418 disk.sda.out‚299.5458984375 disk.rbd0.in‚0 disk.sdb1.out‚0 disk.sda1.in‚0 disk.dm-0.in‚ 2.86328125 disk.sdb.out‚299.5458984375 network.sent‚ 4.671535e+06( disk./.usagePercent‚75.8114543428073% disk./.usageGB‚2711.3848190307617™1—€2ø- _wandbruntime‚2631  _runtime‚2631)Z“†ÇùÁ _step‚183 & lr-AdamW/pg1‚0.0008366384133126595 & lr-AdamW/pg2‚0.0008366384133126595 trainer/global_step‚2299 % _timestamp‚1.7885130064067783e+09 _runtime‚2631.878274691ÌúÆ & lr-AdamW/pg2‚0.0008366384133126595 trainer/global_step‚2299 % _timestamp‚1.7885130064067783e+09 _runtime‚2631.878274691 _step‚183 & lr-AdamW/pg1‚0.0008366384133126595·íFÊ2û- _wandbruntime‚2631  _runtime‚2631†tm=ò'üìO VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚189141540349952 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚29065573 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917409792 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 104022441 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚186550073984512 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚42653626 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917409792 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚208947847634944 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26088 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26086 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2591448362496 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚186550073984512 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚42653037 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2580881291301 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7885130064076664e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31929 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚46738444 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚189141540349952 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚587 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13587464 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚211652646194176 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚182511309278192 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚57257911 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177046016 _step‚184 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 trainer/global_step‚2299 _runtime‚2631.878370991 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899406848 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955590144 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚29065575 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚46738442 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26088 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2704578086912 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚186550091987456 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2590548955648 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1205 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚182511292238832 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411412 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899406848 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚211652401781760 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2704798559232 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31684 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚46738444 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚185093087980905 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚185092173530133 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚46738442 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955590144 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚57257911 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚189140622940160 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177046016 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31684 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚57283997 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26086 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 244412416 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214657 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 103996353 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 398533632 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712Û >PÛýñO WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚57283997 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26086 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 244412416 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214657 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 103996353 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 398533632 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚189141540349952 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚29065573 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917409792 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 104022441 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚186550073984512 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚42653626 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917409792 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚208947847634944 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26088 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26086 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2591448362496 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚186550073984512 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚42653037 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2580881291301 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7885130064076664e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31929 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚46738444 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚189141540349952 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚587 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13587464 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚211652646194176 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚182511309278192 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚57257911 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177046016 _step‚184 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 trainer/global_step‚2299 _runtime‚2631.878370991 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899406848 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955590144 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚29065575 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚46738442 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26088 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2704578086912 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚186550091987456 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2590548955648 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1205 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚182511292238832 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411412 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899406848 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚211652401781760 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2704798559232 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31684 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚46738444 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚185093087980905 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚185092173530133 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚46738442 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955590144 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚57257911 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚189140622940160 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177046016 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31684 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171¸Ã#2þ- _wandbruntime‚2632  _runtime‚2632,Ppû&ÿõM QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 398533632 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚594 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚189221366039040 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26114 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚209037142915584 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 _step‚185 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚189222283462144 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26112 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚57282502 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007233 $ _timestamp‚1.788513007382549e+09 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 104066956 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955590144 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 104066956 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 238107648 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917423104 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚186629722283008 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214657 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899420160 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚185171262285579 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚57308614 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚57282502 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2591661758976 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13593812 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2581989952619 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚186629722283008 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚46758342 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26112 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917423104 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚57308614 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚211742899035648 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2582887364239 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚29077951 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 104040842 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚42671169 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚592 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712AÀI‘'€úM H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 104066956 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955590144 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 104066956 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 238107648 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917423104 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚186629722283008 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214657 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899420160 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚185171262285579 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚57308614 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚57282502 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2591661758976 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13593812 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2581989952619 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚186629722283008 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚46758342 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26112 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917423104 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚57308614 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚211742899035648 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2582887364239 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚29077951 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 104040842 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚42671169 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚592 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 398533632 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚594 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚189221366039040 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26114 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚209037142915584 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 _step‚185 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚189222283462144 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26112 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚57282502 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007233 $ _timestamp‚1.788513007382549e+09¹ÉšäØ2- _wandbruntime‚2632  _runtime‚2632úxªV‚Ð 0loss/FBt1/assignment_loss‚0.2597929303976178 /loss/FBt3/detection_loss‚2.6901637424122202 -loss/FRt2/detection_loss‚2.22301877640626 /loss/FRt1/assignment_loss‚3.597218704778095 /loss/FRt3/assignment_loss‚3.381605880440406 2loss/SRqqt3/assignment_loss‚1.3284043154734022 1loss/SRqqt2/detection_loss‚1.3265365801884659 .loss/FRt3/detection_loss‚2.404088022011631 0loss/FRt4/assignment_loss‚2.9762814657555685 /loss/FRt1/detection_loss‚2.0682889015175845 1loss/FBt2/assignment_loss‚0.17699915218393125 1loss/FBt3/assignment_loss‚0.08889364506057973 0loss/SRqqt4/detection_loss‚2.486601766078703 trainer/global_step‚2299 % _timestamp‚1.7885130073847904e+09 .loss/FBt1/detection_loss‚2.581077293916182 epoch‚1 /loss/FRt2/assignment_loss‚1.861777300343794 2loss/SRqqt2/assignment_loss‚1.1996347724260497 1loss/FBt4/assignment_loss‚0.06467101036091094 _step‚186 2loss/SRqqt1/assignment_loss‚1.4724001301950667 0loss/SRqqt1/detection_loss‚2.199058639795281 1loss/SRqqt3/detection_loss‚1.7576453522139905 .loss/FBt2/detection_loss‚0.686280239246447 .loss/FBt4/detection_loss‚3.147273689508438 %loss/total_loss‚44.90208200062896 _runtime‚2632.855393632 .loss/FRt4/detection_loss‚2.678956985473633 2loss/SRqqt4/assignment_loss‚2.2454127044447008šJ[ƒÕ .loss/FBt2/detection_loss‚0.686280239246447 .loss/FBt4/detection_loss‚3.147273689508438 %loss/total_loss‚44.90208200062896 _runtime‚2632.855393632 .loss/FRt4/detection_loss‚2.678956985473633 2loss/SRqqt4/assignment_loss‚2.2454127044447008 0loss/FBt1/assignment_loss‚0.2597929303976178 /loss/FBt3/detection_loss‚2.6901637424122202 -loss/FRt2/detection_loss‚2.22301877640626 /loss/FRt1/assignment_loss‚3.597218704778095 /loss/FRt3/assignment_loss‚3.381605880440406 2loss/SRqqt3/assignment_loss‚1.3284043154734022 1loss/SRqqt2/detection_loss‚1.3265365801884659 .loss/FRt3/detection_loss‚2.404088022011631 0loss/FRt4/assignment_loss‚2.9762814657555685 /loss/FRt1/detection_loss‚2.0682889015175845 1loss/FBt2/assignment_loss‚0.17699915218393125 1loss/FBt3/assignment_loss‚0.08889364506057973 0loss/SRqqt4/detection_loss‚2.486601766078703 trainer/global_step‚2299 % _timestamp‚1.7885130073847904e+09 .loss/FBt1/detection_loss‚2.581077293916182 epoch‚1 /loss/FRt2/assignment_loss‚1.861777300343794 2loss/SRqqt2/assignment_loss‚1.1996347724260497 1loss/FBt4/assignment_loss‚0.06467101036091094 _step‚186 2loss/SRqqt1/assignment_loss‚1.4724001301950667 0loss/SRqqt1/detection_loss‚2.199058639795281 1loss/SRqqt3/detection_loss‚1.7576453522139905º&‰d4ž„‚(:“ ö•êÔò‘äý gpu.0.gpu‚57.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚119.368' gpu.0.enforcedPowerLimitWatts‚400.0) gpu.0.powerPercent‚29.841999999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0+ÀÙÂQ…‚(:Æ ö•êÔòí€# disk.nvme1n1.out‚13495.87109375 disk.sdb1.out‚0 disk.nvme0n1.in‚1.4375 disk.sdb.out‚ 300.998046875 disk.sda.in‚ 0.0390625* proc.memory.percent‚18.288779258728027 network.sent‚ 4.688453e+06) disk./.usagePercent‚75.81146607546583 disk.md0p1.out‚ 289.90625 disk.sdb1.in‚0 disk.dm-0.in‚ 2.86328125 disk.sdb.in‚0.03125& proc.memory.availableMB‚ 3.18359375# proc.memory.rssMB‚ 2996.43359375 network.recv‚884236 disk.md0.in‚ 0.0703125 disk.rbd2.out‚ 0.783203125 disk.sda1.out‚0$ memory_percent‚99.98056888580322 disk.nvme1n1.in‚ 1.42578125 disk.sdb2.out‚ 300.998046875 disk.dm-0.out‚28655.31640625" disk.nvme0n1.out‚ 15159.4453125 disk.sda.out‚ 300.998046875 disk.rbd0.out‚ 0.78271484375 disk.sda1.in‚0 disk.sda2.in‚ 0.0390625 disk.sda2.out‚ 300.998046875 disk.rbd0.in‚0 proc.cpu.threads‚17$ disk./.usageGB‚2711.385238647461 disk.rbd2.in‚0 disk.sdb2.in‚0.03125 cpu‚25.14503434360205 disk.md0.out‚ 289.90625 disk.md0p1.in‚ 0.0703125µit¾”†‚(:‰ …–êÔлåý gpu.0.gpu‚38.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚166.207' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚41.55175 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0hÂ7L‡‚(:Á …–êÔמ˜€ disk.rbd0.out‚ 0.78271484375 disk.sdb1.in‚0 disk.sdb2.in‚0.03125 disk.md0p1.out‚ 291.91796875" proc.memory.availableMB‚2.9375! disk.nvme1n1.out‚ 13496.140625 network.recv‚888858) disk./.usagePercent‚75.81145348952305 disk.sdb.out‚ 303.0859375 disk.sda.out‚ 303.0859375 disk.sda2.in‚ 0.0390625 disk.sda2.out‚ 303.0859375 disk.md0.out‚ 291.91796875 disk.nvme1n1.in‚ 1.42578125 disk.md0.in‚ 0.0703125 disk.dm-0.out‚28739.95703125 disk.md0p1.in‚ 0.0703125 disk.nvme0n1.in‚1.4375# proc.memory.rssMB‚ 2996.43359375 disk.sdb.in‚0.03125 disk.sda1.in‚0 cpu‚25.144314395290788 proc.cpu.threads‚17% disk./.usageGB‚2711.3847885131836 disk.sda.in‚ 0.0390625 disk.rbd0.in‚0 disk.rbd2.in‚0 disk.sdb2.out‚ 303.0859375 disk.sda1.out‚0 network.sent‚ 4.784749e+06 disk.sdb1.out‚0 disk.rbd2.out‚ 0.783203125 disk.dm-0.in‚ 2.86328125* proc.memory.percent‚18.288779258728027# disk.nvme0n1.out‚15243.81640625$ memory_percent‚99.98207092285156¹Úä唈‚(:‰ ”–êÔæǘþ gpu.0.gpu‚40.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚169.333' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.33325 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¹+‡O‰‚(:Ä ”–êÔ„‡Ô€ disk.dm-0.out‚ 28741.15625 disk.sdb.out‚304.53076171875 disk.sda.out‚304.53076171875# proc.memory.rssMB‚ 2996.43359375! disk.nvme1n1.out‚ 13497.328125 disk.sda1.in‚0 disk.dm-0.in‚ 2.86328125 disk.md0p1.out‚293.3125 proc.cpu.threads‚17% disk./.usageGB‚2711.3676795959473 disk.sdb1.out‚0 disk.sda1.out‚0% proc.memory.availableMB‚ 3.1015625* proc.memory.percent‚18.288779258728027 disk.md0.in‚ 0.0703125 disk.rbd2.in‚0! disk.nvme0n1.out‚ 15243.828125$ memory_percent‚99.98106956481934) disk./.usagePercent‚75.81097511703648! disk.sdb2.out‚304.53076171875 network.sent‚ 4.790652e+06! disk.sda2.out‚304.53076171875 disk.rbd0.out‚ 0.845703125 disk.sdb2.in‚0.03125 disk.sda.in‚ 0.0390625 disk.rbd2.out‚ 0.783203125 disk.md0p1.in‚ 0.0703125 disk.nvme0n1.in‚1.4375 disk.sdb.in‚0.03125 network.recv‚893373 disk.nvme1n1.in‚ 1.42578125 disk.sdb1.in‚0 cpu‚25.14359481466248 disk.sda2.in‚ 0.0390625 disk.rbd0.in‚0 disk.md0.out‚293.3125`mÞ ÀŠ‚(:ˆ £–êÔüäåÿ gpu.0.gpu‚12.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWá¥jÓatts‚175.666' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.9165 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0â<ÒËT‹‚(:É £–êÔÞà‰€ disk.rbd2.out‚ 0.783203125 disk.sdb2.in‚0.03125 disk.sda1.in‚0 disk.dm-0.out‚ 29142.7109375# proc.memory.availableMB‚3.34375 network.sent‚ 4.796633e+06 disk.sdb1.in‚0 proc.cpu.threads‚17 disk.md0p1.out‚ 295.21875# proc.memory.rssMB‚ 2996.43359375 cpu‚25.142996095779427 disk.sdb.in‚0.03125 disk.sda.out‚306.50341796875 disk.md0.in‚ 0.0703125! disk.sda2.out‚306.50341796875 disk.rbd0.in‚0 disk.md0.out‚ 295.21875 disk.sda1.out‚0 disk.nvme0n1.in‚1.4375* proc.memory.percent‚18.288779258728027 network.recv‚897414 disk.rbd0.out‚ 0.845703125 disk.rbd2.in‚0# memory_percent‚99.9795913696289$ disk./.usageGB‚2711.436180114746 disk.sdb.out‚306.50341796875! disk.sdb2.out‚306.50341796875) disk./.usagePercent‚75.81289042021177 disk.sda.in‚ 0.0390625# disk.nvme1n1.out‚13638.97265625 disk.dm-0.in‚ 2.86328125 disk.md0p1.in‚ 0.0703125# disk.nvme0n1.out‚15503.73828125 disk.sda2.in‚ 0.0390625 disk.nvme1n1.in‚ 1.42578125 disk.sdb1.out‚0%d2Œ- _wandbruntime‚2685  _runtime‚2685]{ÕÇÁ & lr-AdamW/pg1‚0.0008346796524722668 & lr-AdamW/pg2‚0.0008346796524722668 trainer/global_step‚2349 % _timestamp‚1.7885130604505432e+09 _runtime‚2685.921140966 _step‚187~UŒ`ÌŽÆ & lr-AdamW/pg1‚0.0008346796524722668 & lr-AdamW/pg2‚0.0008346796524722668 trainer/global_step‚2349 % _timestamp‚1.7885130604505432e+09 _runtime‚2685.921140966 _step‚187»Qx .2- _wandbruntime‚2685  _runtime‚2685ëGºñ'ëO N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917409792 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚193177792297984 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899406848 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177046016 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚47733344 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚43559260 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚29684375 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚47733342 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚186410313984592 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31929 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13875473 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177046016 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 106250243 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚216176969788928 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26086 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚213413784731648 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2636327948409 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚190531606765056 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1180 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007233 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13874887 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚213413808671744 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26088 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2647102942720 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚47733342 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2762942742016 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚186410296945232 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2763161117184 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚189046624893641 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2647102942720 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26086 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚58490813 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31684 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917409792 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚586 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31684 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26088 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955590144 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2637225359821 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚190531606765056 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1205 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411412 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899406848 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚193177792297984 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 trainer/global_step‚2349 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚216176727473664 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚29684373 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 106224155 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955590144 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚193178709707776 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚47733344 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚193178709707776 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2646203535872 _runtime‚2685.922670633 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚58516899 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚29684373 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 106224155 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955590144 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚193178709707776 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚47733344 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚193178709707776 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2646203535872 _runtime‚2685.922670633 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚58516899 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚193177792297984 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899406848 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177046016 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚47733344 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚43559260 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚29684375 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚47733342 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚186410313984592 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31929 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13875473 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177046016 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 106250243 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚216176969788928 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26086 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚213413784731648 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2636327948409 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚190531606765056 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1180 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007233 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13874887 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚213413808671744 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26088 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2647102942720 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚47733342 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2762942742016 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚186410296945232 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2763161117184 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚189046624893641 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2647102942720 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26086 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚58490813 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31684 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917409792 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚586 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31684 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26088 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955590144 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2637225359821 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚190531606765056 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1205 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411412 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899406848 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚193177792297984¼ÞLZþ2’- _wandbruntime‚2686  _runtime‚2686¶rÎCü&“öM OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚605 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚190611237060608 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚193258535396864 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚58541516 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13881144 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 222556160 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2764109805056 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚607 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚29696751 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚13880539 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450980 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2648215759360 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899420160 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 106268644 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚43577895 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚213503103952384 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214657 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚58515404 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚216267213757440 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 106294758 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 106294758 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚43577288 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2647316339200 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899420160 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚29696749 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2647316339200 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚193259452819968 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31684 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007233 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1180 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2637436609727 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚213503127892480 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917423104 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177046016 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955590144 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚186488277039360 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411620 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 _runtime‚2686.896416776 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1205 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚189 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26112 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚190611237060608 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 106268644 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31929 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31929 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917423104 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26112 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 trainer/global_step‚2349 % _timestamp‚1.7885130614251766e+09`7€”ûM VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚29696751 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚13880539 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450980 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_eúÃ%pool.allocated‚ 2648215759360 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899420160 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 106268644 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚43577895 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚213503103952384 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214657 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚58515404 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚216267213757440 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 106294758 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 106294758 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚43577288 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2647316339200 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899420160 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚29696749 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2647316339200 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚193259452819968 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31684 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007233 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1180 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2637436609727 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚213503127892480 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917423104 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177046016 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955590144 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚186488277039360 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411620 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 _runtime‚2686.896416776 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1205 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚189 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26112 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚190611237060608 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 106268644 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31929 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31929 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917423104 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26112 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 trainer/global_step‚2349 % _timestamp‚1.7885130614251766e+09 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚605 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚190611237060608 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚193258535396864 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚58541516 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13881144 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 222556160 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2764109805056 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚607 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0½%0ä2•- _wandbruntime‚2686  _runtime‚2686½¥![–Õ /loss/FRt3/assignment_loss‚2.880787423023811 _runtime‚2686.898029324 0loss/FRt2/assignment_loss‚1.3535295700033505 1loss/FBt4/assignment_loss‚0.10218038703023638 1loss/SRqqt1/detection_loss‚1.8992320256975461 -loss/FRt4/assignment_loss‚3.9457952188594 /loss/FBt3/detection_loss‚2.3683915308543613 /loss/FBt4/detection_loss‚2.8510836482048036 /loss/FRt1/assignment_loss‚3.321919431289037 2loss/SRqqt1/assignment_loss‚1.5081332014205884 2loss/SRqqt3/assignment_loss‚1.8607960703472297 1loss/SRqqt2/detection_loss‚1.0996281212594892 %loss/total_loss‚42.59469226336738 trainer/global_step‚2349 2loss/SRqqt4/assignment_loss‚1.6763747299137788 1loss/FBt3/assignment_loss‚0.06106065130318582 .loss/FRt4/detection_loss‚2.922901730452265 0loss/SRqqt3/detection_loss‚2.002539266832173 /loss/FBt1/detection_loss‚2.6338249891996384 _step‚190 1loss/FBt2/assignment_loss‚0.07000453745116805 /loss/FRt2/detection_loss‚1.5173662273058046 /loss/FBt2/detection_loss‚0.7248665486996093 /loss/FRt1/detection_loss‚2.0174070785443003 .loss/FRt3/detection_loss‚2.467282718572861 1loss/SRqqt4/detection_loss‚2.2146084904670715 1loss/SRqqt2/assignment_loss‚1.068506798973041 2loss/FBt1/assignment_loss‚0.026471867662621662 epoch‚1 % _timestamp‚1.7885130614278533e+09*ÁÄ`—Ú 1loss/FBt4/assignment_loss‚0.10218038703023638 1loss/SRqqt1/detection_loss‚1.8992320256975461 -loss/FRt4/assignment_loss‚3.9457952188594 /loss/FBt3/detection_loss‚2.3683915308543613 /loss/FBt4/detection_loss‚2.8510836482048036 /loss/FRt1/assignment_loss‚3.321919431289037 2loss/SRqqt1/assignment_loss‚1.5081332014205884 2loss/SRqqt3/assignment_loss‚1.8607960703472297 1loss/SRqqt2/detection_loss‚1.0996281212594892 %loss/total_loss‚42.59469226336738 trainer/global_step‚2349 2loss/SRqqt4/assignment_loss‚1.6763747299137788 1loss/FBt3/assignment_loss‚0.06106065130318582 .loss/FRt4/detection_loss‚2.922901730452265 0loss/SRqqt3/detection_loss‚2.002539266832173 /loss/FBt1/detection_loss‚2.6338249891996384 _step‚190 1loss/FBt2/assignment_loss‚0.07000453745116805 /loss/FRt2/detection_loss‚1.5173662273058046 /loss/FBt2/detection_loss‚0.7248665486996093 /loss/FRt1/detection_loss‚2.0174070785443003 .loss/FRt3/detection_loss‚2.467282718572861 1loss/SRqqt4/detection_loss‚2.2146084904670715 1loss/SRqqt2/assignment_loss‚1.068506798973041 2loss/FBt1/assignment_loss‚0.026471867662621662 epoch‚1 % _timestamp‚1.7885130614278533e+09 /loss/FRt3/assignment_loss‚2.880787423023811 _runtime‚2686.898029324 0loss/FRt2/assignment_loss‚1.3535295700033505¾ r”˜‚(:‰ ²–êÔ¼óý gpu.0.gpu‚27.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚37.0 gpu.0.powerWatts‚128.665' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚32.16625 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0–VêéX™‚(:Í ²–êÔø™Ä€ disk.rbd0.out‚ 0.845703125 disk.nvme1n1.out‚ 13643.34375$ memory_percent‚99.97994899749756 network.recv‚902258 disk.rbd2.out‚ 0.783203125 disk.rbd0.in‚0 disk.sdb1.in‚0 disk.dm-0.in‚ 2.86328125* proc.memory.percent‚18.288779258728027 disk.sdb2.out‚307.8662109375 disk.sda1.in‚0 disk.sda.in‚ 0.0390625 cpu‚25.142291948203592$ disk./.usageGB‚2710.912483215332 disk.sdb.out‚307.8662109375 disk.sda2.out‚307.8662109375# proc.memory.rssMB‚ 2996.43359375 disk.md0.in‚ 0.0703125 disk.rbd2.in‚0 disk.md0p1.in‚ 0.0703125# disk.nvme0n1.out‚15552.43359375 disk.sda.out‚307.8662109375 disk.nvme1n1.in‚ 1.42578125 network.sent‚ 4.800708e+06 disk.md0.out‚ 296.51953125 disk.sda1.out‚0 disk.md0p1.out‚ 296.51953125 disk.nvme0n1.in‚1.4375 disk.sda2.in‚ 0.0390625& proc.memory.availableMB‚ 3.28515625 disk.sdb2.in‚0.03125 disk.dm-0.out‚29195.77734375 disk.sdb.in‚0.03125 disk.sdb1.out‚0 proc.cpu.threads‚17) disk./.usagePercent‚75.79824763571999ÎÀxîžš‚(:“ Á–êÔóèÉÿ gpu.0.gpu‚13.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚37.0 gpu.0.powerWatts‚115.634' gpu.0.enforcedPowerLimitWatts‚400.0) gpu.0.powerPercent‚28.908499999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÿŽú_›‚(:Ô Á–êÔÛ°–€ disk.rbd0.out‚ 0.845703125" disk.nvme1n1.out‚ 13695.0390625 disk.md0p1.in‚ 0.0703125 disk.sdb2.in‚0.03125! disk.sdb2.out‚310.08837890625% proc.memory.availableMB‚ 3.5234375$ disk./.usageGB‚2710.989948272705 disk.sdb1.out‚0 disk.dm-0.in‚ 213.4921875$ memory_percent‚99.97849464416504* proc.memory.percent‚18.288779258728027 network.sent‚ 4.909745e+06 disk.rbd0.in‚0 disk.nvme1n1.in‚ 184.96484375 disk.rbd2.in‚0 disk.nvme0n1.in‚ 28.52734375! disk.sda2.out‚310.08837890625 proc.cpu.threads‚17 network.recv‚906788) disk./.usagePercent‚75.80041359114577 disk.rbd2.out‚ 0.783203125 disk.nvme0n1.out‚ 15554.1875 disk.md0.in‚ 0.0703125 disk.md0.out‚ 298.67578125 disk.md0p1.out‚ 298.67578125 disk.sda.in‚ 0.0390625 disk.sda.out‚310.08837890625 disk.sda1.out‚0 disk.sdb.in‚0.03125 disk.sdb.out‚310.08837890625 cpu‚25.14152013672658 disk.sdb1.in‚0 disk.sda1.in‚0 disk.dm-0.out‚ 29249.2265625 disk.sda2.in‚ 0.0390625# proc.memory.rssMB‚ 2996.43359375NÚòä’œ‚(:‡ ЖêÔì·ÿ gpu.0.gpu‚40.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚166.656' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚41.664 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0/ã“š[‚(:Ð ЖêÔÏ disk.sda.in‚ 0.0390625 disk.nvme1n1.in‚ 184.96484375" disk.nvme1n1.out‚ 13795.1953125 disk.md0p1.out‚ 300.078125 disk.rbd0.in‚0 disk.sdb2.in‚0.03125 disk.sdb.in‚0.03125 disk.rbd0.out‚ 0.845703125 disk.rbd2.out‚ 0.783203125% disk./.usageGB‚2711.1899604797363 disk.rbd2.in‚0 disk.sda1.out‚0# proc.memory.rssMB‚ 2996.43359375 network.recv‚911399" disk.nvme0n1.out‚ 15631.3359375 disk.md0.in‚ 0.0703125 disk.dm-0.in‚ 213.4921875 disk.dm-0.out‚ 29426.53125 disk.sdb1.in‚0# memory_percent‚99.9776840209961* proc.memory.percent‚18.288779258728027 cpu‚25.140839815962924 disk.sda.out‚311.56005859375 network.sent‚ 4.913478e+06 disk.nvme0n1.in‚ 28.52734375 disk.sdb.out‚311.56005859375 disk.sda2.in‚ 0.0390625 disk.sda1.in‚0# proc.memory.availableMB‚3.65625 disk.md0p1.in‚ 0.0703125! disk.sda2.out‚311.56005859375 disk.sdb1.out‚0 disk.md0.out‚ 300.078125! disk.sdb2.out‚311.56005859375 proc.cpu.threads‚17) disk./.usagePercent‚75.806006016165989ôšË2ž- _wandbruntime‚2742  _runtime‚2742ôþ¯ÇŸÁ _runtime‚2742.454695248 _step‚191 & lr-AdamW/pg1‚0.0008326363174031328 & lr-AdamW/pg2‚0.0008326363174031328 trainer/global_step‚2399 % _timestamp‚1.7885131169841125e+09Å/Ì Æ & lr-AdamW/pg1‚0.0008326363174031328 & lr-AdamW/pg2‚0.0008326363174031328 trainer/global_step‚2399 % _timestamp‚1.7885131169841125e+09 _runtime‚2742.454695248 _step‚191¿:¦2¡- _wandbruntime‚2742  _runtime‚2742üU  ò'¢ìO A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚48728244 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚194513121542656 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177046016 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2692672016929 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1205 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26086 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2702757522944 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚220701157849600 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 % _timestamp‚1.7885131169852946e+09 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26088 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚59749801 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚44466980 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚30303173 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚197214961655808 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚194513103539712 _step‚192 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚30303175 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚14163160 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚197215879065600 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚194513121542656 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚48728244 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚645 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2702757522944 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 108451957 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2691774605517 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177046016 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2821155085824 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚48728242 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1180 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 256995328 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 398533632 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31684 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚59749801 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚14163805 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚190309318690992 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2821388141056 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚2399 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚48728242 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917409792 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917409792 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚193001076257149 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214657 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411412 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31929 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2701858116096 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31929 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚217879769708544 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚217879745768448 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚44466333 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450772 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007233 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 _runtime‚2742.456099375 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2701858116096 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 233055232 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚193001990707921 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26086î$²^c£ñO YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚197215879065600 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955590144 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚197214961655808 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899406848 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚647 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚194513103539712 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899406848 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31684 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955590144 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚220700900854272 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚59723715 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 108478045 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚48728242 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1180 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 256995328 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 398533632 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31684 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚59749801 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚14163805 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚190309318690992 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2821388141056 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚2399 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚48728242 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917409792 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917409792 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚193001076257149 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214657 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411412 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31929 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2701858116096 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31929 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚217879769708544 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚217879745768448 ICDeviceStatsMonitor.on_tlÌ”rain_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚44466333 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450772 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007233 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 _runtime‚2742.456099375 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2701858116096 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 233055232 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚193001990707921 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26086 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚48728244 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚194513121542656 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177046016 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2692672016929 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1205 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26086 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2702757522944 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚220701157849600 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 % _timestamp‚1.7885131169852946e+09 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26088 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚59749801 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚44466980 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚30303173 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚197214961655808 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚194513103539712 _step‚192 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚30303175 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚14163160 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚197215879065600 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚194513121542656 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚48728244 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚645 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2702757522944 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 108451957 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2691774605517 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177046016 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2821155085824 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 108522560 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚646 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1180 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2702970919424 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚220791624853504 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31684 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚197295704754688 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955590144 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26112 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899420160 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2822535924224 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2692883266835 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚59748306 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚30315549 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚193080165012595 trainer/global_step‚2399 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚44485116 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚30315551 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 398533632 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚59748306 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2822311270912 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 108522560 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚48748142 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1205 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚194592751838208 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 108496446 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917423104 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚194592733835264 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚648 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚14169565 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214657 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚190387281745760 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 248593408 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31929 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚44484468 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 % _timestamp‚1.7885131181478405e+09 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917423104 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899420160 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2693780678455 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007233 _runtime‚2743.620476534 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31929SHA'¦ûM ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31684 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚197295704754688 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955590144 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26112 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899420160 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2822535924224 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2692883266835 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚59748306 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚30315549 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚193080165012595 trainer/global_step‚2399 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚44485116 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚30315551 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 398533632 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚59748306 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2822311270912 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 108522560 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚48748142 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1205 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚194592751838208 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 108496446 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917423104 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚194592733835264 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚648 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚14169565 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214657 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚190387281745760 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 248593408 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31929 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚44484468 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 % _timestamp‚1.7885131181478405e+09 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917423104 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899420160 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2693780678455 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007233 _runtime‚2743.620476534 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31929 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚194592733835264 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚59774418 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚220791376260096 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚193081079463575 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚190387298785120 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26114 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚217969064989184 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚48748142 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 108522560 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚646 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1180 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2702970919424 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚220791624853504Á"ßô…2§- _wandbruntime‚2743  _runtime‚2743†Š8_¨Ù .loss/FBt1/detection_loss‚2.383833721280098 0loss/FRt1/assignment_loss‚2.7813621628284455 1loss/SRqqt4/detection_loss‚2.5814223361737803 0loss/SRqqt2/detection_loss‚1.134665551567168 .loss/FBt4/detection_loss‚2.529907214641571 /loss/FRt4/assignment_loss‚4.206304576085961 2loss/SRqqt2/assignment_loss‚0.9557414932148249 1loss/FBt2/assignment_loss‚0.09386268482860405 /loss/FRt3/detection_loss‚2.2503487679693435 2loss/SRqqt4/assignment_loss‚1.7924047766084021 /loss/FBt3/detection_loss‚2.4779129028320312 _step‚194 2loss/FBt3/assignment_loss‚0.019172074946254725 1loss/FBt4/assignment_loss‚0.14337759343907236 epoch‚1 trainer/global_step‚2399 % _timestamp‚1.7885131181514153e+09 _runtime‚2743.621756169 .loss/FBt2/detection_loss‚0.897125284132712 %loss/total_loss‚41.96728067128437 /loss/FRt2/detection_loss‚1.9126172879710794 2loss/SRqqt3/assignment_loss‚1.1977227781645277 2loss/FBt1/assignment_loss‚0.014670991658931598 .loss/FRt4/detection_loss‚3.179313576739767 0loss/FRt2/assignment_loss‚1.7769961267709733 0loss/FRt3/assignment_loss‚2.4113378643989565 /loss/FRt1/detection_loss‚2.1654993855953215 1loss/SRqqt3/detection_loss‚1.5848508089780808 2loss/SRqqt1/assignment_loss‚1.7063200774862142 1loss/SRqqt1/detection_loss‚1.7705106329722482uâªÄd©Þ .loss/FBt2/detection_loss‚0.897125284132712 %loss/total_loss‚41.96728067128437 /loss/FRt2/detection_loss‚1.9126172879710794 2loss/SRqqt3/assignment_loss‚1.1977227781645277 2loss/FBt1/assignment_loss‚0.014670991658931598 .loss/FRt4/detection_loss‚3.179313576739767 0loss/FRt2/assignment_loss‚1.7769961267709733 0loss/FRt3/assignment_loss‚2.4113378643989565 /loss/FRt1/detection_loss‚2.1654993855953215 1loss/SRqqt3/detection_loss‚1.5848508089780808 2loss/SRqqt1/assignment_loss‚1.7063200774862142 1loss/SRqqt1/detection_loss‚1.7705106329722482 .loss/FBt1/detection_loss‚2.383833721280098 0loss/FRt1/assignment_loss‚2.7813621628284455 1loss/SRqqt4/detection_loss‚2.5814223361737803 0loss/SRqqt2/detection_loss‚1.134665551567168 .loss/FBt4/detection_loss‚2.529907214641571 /loss/FRt4/assignment_loss‚4.206304576085961 2loss/SRqqt2/assignment_loss‚0.9557414932148249 1loss/FBt2/assignment_loss‚0.09386268482860405 /loss/FRt3/detection_loss‚2.2503487679693435 2loss/SRqqt4/assignment_loss‚1.7924047766084021 /loss/FBt3/detection_loss‚2.4779129028320312 _step‚194 2loss/FBt3/assignment_loss‚0.019172074946254725 1loss/FBt4/assignment_loss‚0.14337759343907236 epoch‚1 trainer/global_step‚2399 % _timestamp‚1.7885131181514153e+09 _runtime‚2743.621756169»ÁG’ª‚(:‡ ß–êÔûÁ˜€ gpu.0.gpu‚56.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚39.0 gpu.0.powerWatts‚179.856' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚44.964 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Y,ìL]«‚(:Ò ß–êÔó³Ð€ network.sent‚ 4.918574e+06 network.recv‚914675# disk.nvme1n1.out‚13945.44921875 disk.sdb1.in‚0 disk.sda1.in‚0 disk.md0p1.out‚ 301.63671875 disk.sdb.out‚ 313.169921875% disk./.usageGB‚2711.5047302246094 disk.sda2.out‚ 313.169921875 disk.rbd2.in‚0 disk.sdb2.out‚ 313.169921875 disk.md0.out‚ 301.63671875$ memory_percent‚99.97918605804443* proc.memory.percent‚18.288779258728027 proc.cpu.threads‚17 disk.sda.out‚ 313.169921875 disk.sda2.in‚ 0.0390625 disk.rbd0.in‚0 disk.rbd2.out‚ 0.783203125 disk.sdb2.in‚0.03125& proc.memory.availableMB‚ 3.41015625 disk.nvme1n1.in‚ 184.96484375 disk.sdb.in‚0.03125# proc.memory.rssMB‚ 2996.43359375' disk./.usagePercent‚75.814807109974 disk.rbd0.out‚ 0.845703125 disk.sdb1.out‚0 disk.md0.in‚ 0.0703125 disk.md0p1.in‚ 0.0703125 disk.nvme0n1.in‚ 28.52734375 disk.sda.in‚ 0.0390625 disk.sda1.out‚0 disk.dm-0.in‚ 213.4921875 disk.dm-0.out‚29608.51953125" disk.nvme0n1.out‚ 15663.0703125 cpu‚25.140249428127888C3V’¬‚(:‡ î–êÔ»½ÿ gpu.0.gpu‚22.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚37.0 gpu.0.powerWatts‚114.408' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.602 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0•™$ÖQ­‚(:Æ î–êÔüׯ€ disk.dm-0.in‚ 213.4921875 disk.dm-0.out‚ 29766.46875 disk.sdb.in‚0.03125 disk.sdb.out‚ 315.1328125& proc.memory.availableMB‚ 3.16015625 disk.rbd0.out‚ 0.845703125" disk.nvme1n1.out‚ 14045.6328125 disk.rbd2.in‚0 disk.nvme1n1.in‚ 184.96484375 disk.md0.in‚ 0.0703125 network.sent‚ 5.028482e+06 disk.sdb2.in‚0.03125 disk.sda1.out‚0 disk.sda.out‚ 315.1328125 disk.nvme0n1.in‚ 28.52734375 disk.sda2.out‚ 315.1328125# proc.memory.rssMB‚ 2996.43359375) disk./.usagePercent‚75.82005800797036 disk.sdb2.out‚ 315.1328125 disk.md0p1.in‚ 0.0703125 disk.sda2.in‚ 0.0390625 disk.sdb1.out‚0 disk.rbd2.out‚ 0.783203125* proc.memory.percent‚18.288779258728027 proc.cpu.threads‚17 disk.md0p1.out‚ 303.53515625" disk.nvme0n1.out‚ 15720.8359375 disk.sda.in‚ 0.0390625 disk.rbd0.in‚0 disk.sdb1.in‚0 network.recv‚920354$ disk./.usageGB‚2711.692527770996 disk.md0.out‚ 303.53515625$ memory_percent‚99.98071193695068 cpu‚25.13967865017031 disk.sda1.in‚0(®!S”®‚(:‰ ý–êÔ†™–€ gpu.0.gpu‚64.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚37.0 gpu.0.powerWatts‚180.805' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚45.20125 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ê%Ø]¯‚(:Ò ý–êԜ! disk.sdb2.out‚316.77978515625! disk.nvme0n1.out‚ 15808.484375 disk.sdb.out‚316.77978515625) disk./.usagePercent‚75.82706219179373 disk.sda1.in‚0 disk.sda1.out‚0 disk.md0p1.in‚ 0.0703125! disk.sda2.out‚316.77978515625 network.recv‚923702$ disk./.usageGB‚2711.943031311035 disk.md0.in‚ 0.0703125 disk.md0p1.out‚ 305.12109375 disk.nvme0n1.in‚ 28.52734375# proc.memory.rssMB‚ 2996.43359375 disk.sdb2.in‚0.03125 disk.sda.out‚316.77978515625 network.sent‚ 5.03375e+06 disk.rbd0.out‚ 0.845703125 disk.nvme1n1.in‚ 184.96484375 disk.md0.out‚ 305.12109375 disk.sda.in‚ 0.0390625$ proc.memory.availableMB‚3.734375 proc.cpu.threads‚17 disk.rbd0.in‚0 disk.sdb1.out‚0 disk.dm-0.out‚ 29959.71875 disk.sdb.in‚0.03125 disk.sda2.in‚ 0.0390625 cpu‚25.152228974832642 disk.rbd2.out‚ 0.783203125$ memory_percent‚99.97720718383789! disk.nvme1n1.out‚ 14151.234375 disk.sdb1.in‚0 disk.rbd2.in‚0 disk.dm-0.in‚ 213.4921875* proc.memory.percent‚18.288779258728027´ô‡`#°‚(:‡ Œ—êÔ‰ùÞÿ gpu.0.@š•ogpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚36.0 gpu.0.powerWatts‚113.466' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.3665 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0löK€\±‚(:Ñ Œ—êÔßÈ×€ network.recv‚928051 disk.sda.out‚318.37841796875 disk.sdb.in‚0.03125* proc.memory.percent‚18.288779258728027 disk.sdb1.out‚0% disk./.usageGB‚2711.9420127868652! disk.sda2.out‚318.37841796875 disk.sdb1.in‚0 disk.dm-0.in‚ 213.4921875 disk.sda.in‚ 0.0390625 disk.rbd2.out‚ 0.783203125 disk.md0p1.out‚ 306.65625 disk.sdb.out‚318.37841796875 proc.cpu.threads‚17 network.sent‚ 5.037549e+06) disk./.usagePercent‚75.82703371343165 disk.md0.out‚ 306.65625 disk.rbd0.in‚0 disk.nvme1n1.in‚ 184.96484375 disk.rbd2.in‚0 disk.md0p1.in‚ 0.0703125 disk.nvme0n1.out‚ 15923.9375 disk.rbd0.out‚ 0.845703125# disk.nvme1n1.out‚14247.35546875! disk.sdb2.out‚318.37841796875 disk.sda1.in‚0 disk.sda1.out‚0 disk.sdb2.in‚0.03125 disk.nvme0n1.in‚ 28.52734375$ proc.memory.availableMB‚3.484375 cpu‚25.170861875563226 disk.sda2.in‚ 0.0390625 disk.md0.in‚ 0.0703125 disk.dm-0.out‚30171.29296875$ memory_percent‚99.97873306274414# proc.memory.rssMB‚ 2996.43359375¨PÀ2’²‚(:‡ ›—êÔê÷½ÿ gpu.0.gpu‚3.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚35.0 gpu.0.powerWatts‚113.466' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.3665 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0È•0‰f³‚(:Û ›—êԹ貀 network.sent‚ 5.045563e+06 disk.sda.out‚319.93212890625! disk.sda2.out‚319.93212890625 disk.sdb1.out‚0 disk.md0.in‚ 0.0703125 disk.rbd2.in‚0 disk.sda1.in‚0 disk.md0p1.in‚ 0.0703125 disk.sda2.in‚ 0.0390625& proc.memory.availableMB‚ 3.69921875# proc.memory.rssMB‚ 2996.43359375 disk.nvme1n1.in‚ 184.96484375 disk.dm-0.out‚30171.69140625* proc.memory.percent‚18.288779258728027 proc.cpu.threads‚17 network.recv‚932305 disk.sda.in‚ 0.0390625 disk.rbd0.in‚0 disk.sda1.out‚0 disk.nvme0n1.in‚ 28.52734375 disk.rbd0.out‚ 0.845703125) disk./.usagePercent‚75.82703371343165% disk./.usageGB‚2711.9420127868652# disk.nvme0n1.out‚15924.14453125 disk.sdb.in‚0.03125 disk.md0.out‚ 308.14453125 disk.dm-0.in‚ 213.4921875 cpu‚25.189481950937683 disk.sdb.out‚319.93212890625! disk.nvme1n1.out‚ 14247.546875 disk.sdb1.in‚0 disk.rbd2.out‚ 0.783203125! disk.sdb2.out‚319.93212890625 disk.sdb2.in‚0.03125 disk.md0p1.out‚ 308.14453125$ memory_percent‚99.97742176055908qãWÂ’´‚(:‡ ª—êÔãû˜þ gpu.0.gpu‚2.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚35.0 gpu.0.powerWatts‚113.466' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.3665 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0[‹­ãYµ‚(:Î ª—êÔ€ïŒÿ disk.nvme0n1.in‚ 28.52734375$ memory_percent‚99.98044967651367 disk.rbd0.in‚0 disk.dm-0.in‚ 213.4921875 disk.md0p1.in‚ 0.0703125$ proc.memory.availableMB‚3.203125 cpu‚25.21029689903271! disk.nvme1n1.out‚ 14547.640625 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.md0p1.out‚ 310.171875 disk.sdb.out‚322.03076171875 disk.sda.out‚322.03076171875! disk.sda2.out‚322.03076171875) disk./.usagePercent‚75.82703371343165 disk.rbd2.in‚0 disk.sda.in‚ 0.0390625 disk.sdb.in‚0.03125 network.sent‚ 5.049638e+06% disk./.usageGB‚2711.9420127868652 disk.rbd2.out‚ 0.783203125 disk.sdb2.in‚0.03125 disk.dm-0.out‚ 30673.8125 disk.sda2.in‚ 0.0390625# proc.memory.rssMB‚ 2996.43359375* proc.memory.percent‚18.288779258728027 disk.nvme1n1.in‚ 184.96484375 disk.md0.in‚ 0.0703125 disk.md0.out‚ 310.171875 disk.sda1.in‚0 disk.sda1.out‚0! disk.nvme0n1.out‚ 16126.171875 network.recv‚936885 disk.rbd0.out‚ 0.845703125! disk.sdb2.out‚322.03076171875 proc.cpu.threads‚17Aâ’š2¶- _wandbruntime‚2824  _runtime‚2824t Y*·ÓT ZFDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.freed‚201398254112768 I?DeviceStatsMonitor.on_validation_batch_start/active.all.current‚20519 _MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.freed‚ 2747182444544 PJDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.current‚2 GADeviceStatsMonitor.on_validation_batch_start/num_sync_all_streams‚2 `PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 MCDeviceStatsMonitor.on_validation_batch_start/active.small_pool.peak‚31684 ZKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 PGDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.peak‚3677 MDDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.peak‚1205 _PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 % _timestamp‚1.7885131991169846e+09 YIDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.peak‚ 12835443712 QGDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.peak‚31684 aTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.current‚23940096 ICDeviceStatsMonitor.on_validation_batch_start/oversize_segments.peak‚0 OGDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.current‚607 QIDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.allocated‚564 KEDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.freed‚0 K=DeviceStatsMonitor.on_validation_batch_start/active.all.freed‚ 110481137 TJDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.current‚20517 OGDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.current‚188 \LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.peak‚ 12392107008 cQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.allocated‚ 2737614448227 QIDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.allocated‚607 [MDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.current‚ 158325248 XJDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.peak‚ 398533632 dRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.freed‚ 2867612124672 YIDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.allocated‚ 14220787712 OGDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.current‚564 OADeviceStatsMonitor.on_validation_batch_start/active.all.allocated‚ 110501656 WCDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.freed‚201398254112768 UEDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.peak‚ 13510214657 LCDeviceStatsMonitor.on_validation_batch_start/active.large_pool.peak‚3677 LFDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.peak‚0 aMDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.freed‚194229657841440 TDDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.peak‚ 14220787712 `NDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.allocated‚ 2747857560064 cODeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.allocated‚225161605540352 bTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.current‚ 134385152 [KDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.freed‚0 SEDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.current‚ 693118464 [LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.peak‚ 1177046016 J@DeviceStatsMonitor.on_validation_batch_start/allocation.all.peak‚31929 MGDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.freed‚0 ^JDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.allocated‚196967289329027 D>DeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 WGDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.current‚ 14220787712 LFDeviceStatsMonitor.on_validation_batch_start/active.large_pool.current‚2 VHDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.current‚ 693118464 \LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 VNDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.current‚186 aMDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.freed‚198651071668224 _KDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.freed‚225161447215104 hVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.allocated‚ 2867746509824 ]NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.current‚ 1182793728 I@DeviceStatsMonitor.on_validation_batch_start/segment.all.current‚1171 LDDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.peak‚607 QDDeviceStatsMonitor.on_validation_batch_start/active.small_pool.freed‚60626309 QDDeviceStatsMonitor.on_validation_batch_start/active.large_pool.freed‚49854828 MCDeviceStatsMonitor.on_validation_batch_start/allocation.all.current‚20519 OADeviceStatsMonitor.on_validation_batch_start/allocation.all.freed‚ 110481137 _MDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.freed‚ 2736941303475 REDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.freed‚45541591 E?DeviceStatsMonitor.on_validation_batch_start/num_oom_rejections‚0 QKDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.allocated‚0 VHDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.current‚ 690184112 ^NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.current‚ 13037993984 \ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.current‚18002944 cQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.allocated‚ 2747857560064 SEDeviceStatsMonitor.on_validation_batch_start/allocation.all.allocated‚ 110501656 RKDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.peak‚35 YLDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.current‚18002944 XIDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.peak‚ 1177046016 UEDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.peak‚ 13955590144 TKDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.peak‚1180 ZFDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.freed‚196966599144915 trainer/global_step‚2417 _step‚195 RBDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.peak‚ 13955590144 bNDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.allocated‚198651089671168 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.freed‚31000877 eQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.allocated‚194229674880800 JDDeviceStatsMonitor.on_validation_batch_start/oversize_segments.freed‚0 KBDeviceStatsMonitor.on_validation_batch_start/segment.all.allocated‚1171 \JDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.freed‚ 2747182444544 jVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.allocated‚222293859030528 UHDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.freed‚49854828 UHDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.freed‚60626309 TNDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.current‚2 ^JDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.freed‚198651071668224 ;5DeviceStatsMonitor.on_validation_batch_start/num_ooms‚0 ZLDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.current‚ 675115520 YLDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.allocated‚49854830 VIDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.allocated‚45541779 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 B;DeviceStatsMonitor.on_validation_batch_start/max_split_size‚-1 ]ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.current‚ 675115520 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.freed‚14540714 OIDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 \ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.peak‚564 [GDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.allocated‚201398947231232 PFDeviceStatsMonitor.on_validation_batch_start/active.small_pool.current‚20517 LFDeviceStatsMonitor.on_validation_batch_start/oversize_segments.current‚0 F=DeviceStatsMonitor.on_validation_batch_start/segment.all.peak‚1171 UHDeviceStatsMonitor.on_validation_batch_start/active.large_pool.allocated‚49854830 eQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.allocated‚198651089671168 YLDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.allocated‚60646826 _runtime‚2824.587782214 KEDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.freed‚0 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.allocated‚31000879 ^JDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.allocated‚201398947231232 F=DeviceStatsMonitor.on_validation_batch_start/num_device_alloc‚1171 ]ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.current‚ 673144752 KEDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.freed‚0 [LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.peak‚ 1175007233 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.freed‚0 FéÛ^*¸ØT ZLDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.current‚ 675115520 YLDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.allocated‚49854830 VIDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.allocated‚45541779 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 B;DeviceStatsMonitor.on_validation_batch_start/max_split_size‚-1 ]ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.current‚ 675115520 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.freed‚14540714 OIDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 \ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.peak‚564 [GDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.allocated‚201398947231232 PFDeviceStatsMonitor.on_validation_batch_start/active.small_pool.current‚20517 LFDeviceStatsMonitor.on_validation_batch_start/oversize_segments.current‚0 F=DeviceStatsMonitor.on_validation_batch_start/segment.all.peak‚1171 UHDeviceStatsMonitor.on_validation_batch_start/active.large_pool.allocated‚49854830 eQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.allocated‚198651089671168 YLDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.allocated‚60646826 _runtime‚2824.587782214 KEDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.freed‚0 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.allocated‚31000879 ^JDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.allocated‚201398947231232 F=DeviceStatsMonitor.on_validation_batch_start/num_device_alloc‚1171 ]ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.current‚ 673144752 KEDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.freed‚0 [LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.peak‚ 1175007233 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.freed‚0 FDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 WGDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.current‚ 14220787712 LFDeviceStatsMonitor.on_validation_batch_start/active.large_pool.current‚2 VHDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.current‚ 693118464 \LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 VNDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.current‚186 aMDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.freed‚198651071668224 _KDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.freed‚225161447215104 hVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.allocated‚ 2867746509824 ]NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.current‚ 1182793728 I@DeviceStatsMonitor.on_validation_batch_start/segment.all.current‚1171 LDDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.peak‚607 QDDeviceStatsMonitor.on_validation_batch_start/active.small_pool.freed‚60626309 QDDeviceStatsMonitor.on_validation_batch_start/active.large_pool.freed‚49854828 MCDeviceStatsMonitor.on_validation_batch_start/allocation.all.current‚20519 OADeviceStatsMonitor.on_validation_batch_start/allocation.all.freed‚ 110481137 _MDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.freed‚ 2736941303475 REDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.freed‚45541591 E?DeviceStatsMonitor.on_validation_batch_start/num_oom_rejections‚0 QKDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.allocated‚0 VHDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.current‚ 690184112 ^NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.current‚ 13037993984 \ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.current‚18002944 cQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.allocated‚ 2747857560064 SEDeviceStatsMonitor.on_validation_batch_start/allocation.all.allocated‚ 110501656 RKDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.peak‚35 YLDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.current‚18002944 XIDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.peak‚ 1177046016 UEDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.peak‚ 13955590144 TKDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.peak‚1180 ZFDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.freed‚196966599144915 trainer/global_step‚2417 _step‚195 RBDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.peak‚ 13955590144 bNDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.allocated‚198651089671168 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.freed‚31000877 eQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.allocated‚194229674880800 JDDeviceStatsMonitor.on_validation_batch_start/oversize_segments.freed‚0 KBDeviceStatsMonitor.on_validation_batch_start/segment.all.allocated‚1171 \JDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.freed‚ 2747182444544 jVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.allocated‚222293859030528 UHDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.freed‚49854828 UHDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.freed‚60626309 TNDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.current‚2 ^JDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.freed‚198651071668224 ;5DeviceStatsMonitor.on_validation_batch_start/num_ooms‚0ÃtE32¹- _wandbruntime‚2825  _runtime‚2825€vÙ«êºÝR `LDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.allocated‚198704660946944 cODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.allocated‚198704660946944 [MDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.current‚ 675115520 SFDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.freed‚49870104 PIDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.peak‚35 HBDeviceStatsMonitor.on_validation_batch_end/oversize_segments.freed‚0 trainer/global_step‚2417 @9DeviceStatsMonitor.on_validation_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_validation_batch_end/active_bytes.all.freed‚201452313951232 VGDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.peak‚ 1177046016 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.freed‚31010317 ]IDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.freed‚225217385249792 H>DeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚31929 SCDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.peak‚ 13510214657 G=DeviceStatsMonitor.on_validation_batch_end/active.all.current‚20519 RHDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.current‚20517 TGDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.allocated‚45556596 KBDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.peak‚1205 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.freed‚14546090 YKDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.current‚ 158325248 KEDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.freed‚0 JDDeviceStatsMonitor.on_validation_batch_end/active.large_pool.current‚2 OBDeviceStatsMonitor.on_validation_batch_end/active.large_pool.freed‚49870104 NDDeviceStatsMonitor.on_validation_batch_end/active.small_pool.current‚20517 `RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.current‚ 134385152 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.freed‚0 JADeviceStatsMonitor.on_validation_batch_end/active.large_pool.peak‚3677 WGDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.allocated‚ 14220787712 RBDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.peak‚ 14220787712 P@DeviceStatsMonitor.on_validation_batch_end/active_bytes.all.peak‚ 13955590144 XIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 SFDeviceStatsMonitor.on_validation_batch_end/active.small_pool.allocated‚60655600 RIDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.peak‚1180 JDDeviceStatsMonitor.on_validation_batch_end/oversize_segments.current‚0 UEDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.current‚ 14220787712 MEDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.current‚607 ZMDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.current‚17039360 YIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.freed‚0 RLDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.current‚2 YEDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.allocated‚201453007069696 SFDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.freed‚60635083 fTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.allocated‚ 2868235072512 bPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.freed‚ 2868100687360 M?DeviceStatsMonitor.on_validation_batch_end/allocation.all.freed‚ 110505187 NHDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.current‚2 MEDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.current‚189 D;DeviceStatsMonitor.on_validation_batch_end/num_device_alloc‚1171 ICDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.peak‚607 JBDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.peak‚564 ZJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.peak‚ 12392107008 _runtime‚2825.628669127 \HDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.freed‚198704642944000 C=DeviceStatsMonitor.on_validation_batch_end/num_oom_rejections‚0 cODeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.allocated‚194279444970624 [LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.current‚ 1182793728 D;DeviceStatsMonitor.on_validation_batch_end/segment.all.peak‚1171 ICDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.freed‚0 KADeviceStatsMonitor.on_validation_batch_end/allocation.all.current‚20519 BDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚1171 YJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.peak‚ 1177046016 @:DeviceStatsMonitor.on_validation_batch_end/num_device_free‚0 \HDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.allocated‚197017546345575 % _timestamp‚1.7885132001577992e+09 WJDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.current‚18002944 XDDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.freed‚201452313951232 VHDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.peak‚ 398533632 TFDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.current‚ 690184112 OEDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.peak‚31684 hTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.allocated‚222349308502528 [MDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.current‚ 673144752 M?DeviceStatsMonitor.on_validation_batch_end/active.all.allocated‚ 110525706 \HDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.allocated‚201453007069696 ]KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.freed‚ 2747671007232 QCDeviceStatsMonitor.on_validation_batch_end/allocation.all.allocated‚ 110525706 _RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.current‚23940096 MGDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.current‚0 YJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.peak‚ 1175007233 WGDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.peak‚ 12835443712 WJDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.allocated‚49870106 aMDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.allocated‚225217543575040 LFDeviceStatsMonitor.on_validation_batch_end/oversize_segments.allocated‚0 93DeviceStatsMonitor.on_validation_batch_end/num_ooms‚0 XJDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.current‚ 675115520 ZHDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.freed‚ 2747671007232 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 NEDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.peak‚3677 PCDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.freed‚45556407 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.allocated‚31010319 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.allocated‚14546277 XDDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.freed‚197016856161463 ICDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_validation_batch_end/oversize_segments.peak‚0 OGDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.allocated‚607 OGDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.allocated‚564 I;DeviceStatsMonitor.on_validation_batch_end/active.all.freed‚ 110505187 D:DeviceStatsMonitor.on_validation_batch_end/active.all.peak‚31929 TFDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.current‚ 693118464 OIDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.allocated‚0 ^NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 E?DeviceStatsMonitor.on_validation_batch_end/num_sync_all_streams‚2 _KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.freed‚194279427931264 I@DeviceStatsMonitor.on_validation_batch_end/segment.all.allocated‚1171 _KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.freed‚198704642944000 dPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.freed‚222349284562432#º8Äh)»âR M?DeviceStatsMonitor.on_validation_batch_end/active.all.allocated‚ 110525706 \HDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.allocated‚201453007069696 ]KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.freed‚ 2747671007232 QCDeviceStatsMonitor.on_validation_batch_end/allocation.all.allocated‚ 110525706 _RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.current‚23940096 MGDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.current‚0 YJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.peak‚ 1175007233 WGDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.peak‚ 12835443712 WJDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.allocated‚49870106 aMDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.allocated‚225217543575040 LFDeviceStatsMonitor.on_validation_batch_end/oversize_segments.allocated‚0 93DeviceStatsMonitor.on_validation_batch_end/num_ooms‚0 XJDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.current‚ 675115520 ZHDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.freed‚ 2747671007232 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 NEDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.peak‚3677 PCDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.freed‚45556407 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.allocated‚31010319 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.allocated‚14546277 XDDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.freed‚197016856161463 ICDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_validation_batch_end/oversize_segments.peak‚0 OGDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.allocated‚607 OGDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.allocated‚564 I;DeviceStatsMonitor.on_validation_batch_end/active.all.freed‚ 110505187 D:DeviceStatsMonitor.on_validation_batch_end/active.all.peak‚31929 TFDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.current‚ 693118464 OIDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.allocated‚0 ^NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 E?DeviceStatsMonitor.on_validation_batch_end/num_sync_all_streams‚2 _KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.freed‚194279427931264 I@DeviceStatsMonitor.on_validation_batch_end/segment.all.allocated‚1171 _KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.freed‚198704642944000 dPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.freed‚222349284562432 `LDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.allocated‚198704660946944 cODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.allocated‚198704660946944 [MDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.current‚ 675115520 SFDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.freed‚49870104 PIDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.peak‚35 HBDeviceStatsMonitor.on_validation_batch_end/oversize_segments.freed‚0 trainer/global_step‚2417 @9DeviceStatsMonitor.on_validation_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_validation_batch_end/active_bytes.all.freed‚201452313951232 VGDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.peak‚ 1177046016 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.freed‚31010317 ]IDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.freed‚225217385249792 H>DeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚31929 SCDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.peak‚ 13510214657 G=DeviceStatsMonitor.on_validation_batch_end/active.all.current‚20519 RHDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.current‚20517 TGDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.allocated‚45556596 KBDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.peak‚1205 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.freed‚14546090 YKDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.current‚ 158325248 KEDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.freed‚0 JDDeviceStatsMonitor.on_validation_batch_end/active.large_pool.current‚2 OBDeviceStatsMonitor.on_validation_batch_end/active.large_pool.freed‚49870104 NDDeviceStatsMonitor.on_validation_batch_end/active.small_pool.current‚20517 `RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.current‚ 134385152 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.freed‚0 JADeviceStatsMonitor.on_validation_batch_end/active.large_pool.peak‚3677 WGDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.allocated‚ 14220787712 RBDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.peak‚ 14220787712 P@DeviceStatsMonitor.on_validation_batch_end/active_bytes.all.peak‚ 13955590144 XIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 SFDeviceStatsMonitor.on_validation_batch_end/active.small_pool.allocated‚60655600 RIDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.peak‚1180 JDDeviceStatsMonitor.on_validation_batch_end/oversize_segments.current‚0 UEDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.current‚ 14220787712 MEDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.current‚607 ZMDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.current‚17039360 YIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.freed‚0 RLDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.current‚2 YEDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.allocated‚201453007069696 SFDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.freed‚60635083 fTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.allocated‚ 2868235072512 bPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.freed‚ 2868100687360 M?DeviceStatsMonitor.on_validation_batch_end/allocation.all.freed‚ 110505187 NHDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.current‚2 MEDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.current‚189 D;DeviceStatsMonitor.on_validation_batch_end/num_device_alloc‚1171 ICDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.peak‚607 JBDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.peak‚564 ZJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.peak‚ 12392107008 _runtime‚2825.628669127 \HDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.freed‚198704642944000 C=DeviceStatsMonitor.on_validation_batch_end/num_oom_rejections‚0 cODeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.allocated‚194279444970624 [LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.current‚ 1182793728 D;DeviceStatsMonitor.on_validation_batch_end/segment.all.peak‚1171 ICDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.freed‚0 KADeviceStatsMonitor.on_validation_batch_end/allocation.all.current‚20519 BDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚1171 YJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.peak‚ 1177046016 @:DeviceStatsMonitor.on_validation_batch_end/num_device_free‚0 \HDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.allocated‚197017546345575 % _timestamp‚1.7885132001577992e+09 WJDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.current‚18002944 XDDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.freed‚201452313951232 VHDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.peak‚ 398533632 TFDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.current‚ 690184112 OEDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.peak‚31684 hTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.allocated‚222349308502528 [MDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.current‚ 673144752ÄÖÐÝѼ‚(:… ¹—êÔ¹–øý gpu.0.gpu‚62.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚37.0 gpu.0.powerWatts‚125.88' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚31.47 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0úû ªP½‚(:Å ¹—êÔý¡‚€ disk.sda2.out‚ 323.56640625 disk.nvme1n1.out‚ 14548.125 disk.rbd2.out‚ 0.783203125# disk.nvme0n1.out‚16126.72265625 disk.sda.in‚ 0.0390625 network.recv‚940707( disk./.usagePercent‚75.7954375573407 disk.sdb2.out‚ 323.56640625 disk.sdb.out‚ 323.56640625* proc.memory.percent‚18.288779258728027 disk.nvme1n1.in‚ 184.96484375 disk.sda1.out‚0% proc.memory.availableMB‚ 3.4140625 cpu‚25.233649117504132 disk.sdb.in‚0.03125 network.sent‚ 5.055033e+06$ disk./.usageGB‚2710.811981201172 disk.rbd0.out‚ 0.845703125 disk.nvme0n1.in‚ 28.52734375# proc.memory.rssMB‚ 2996.43359375 disk.md0.in‚ 0.0703125 disk.md0.out‚ 311.640625 disk.sda1.in‚0 disk.dm-0.out‚30674.84765625 disk.md0p1.in‚ 0.0703125 disk.sda.out‚ 323.56640625$ memory_percent‚99.97916221618652 disk.rbd0.in‚0 disk.sdb1.out‚0 disk.rbd2.in‚0 disk.dm-0.in‚ 213.4921875 disk.sda2.in‚ 0.0390625 disk.sdb1.in‚0 disk.sdb2.in‚0.03125 disk.md0p1.out‚ 311.640625 proc.cpu.threads‚17GTu2¾- _wandbruntime‚2839  _runtime‚28399‹oÅã?¿ô§ @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 +jet/accuracy_2_of_3‚0.24936501681804657 8"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.949999988079071 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 :"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.49845677614212036 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 (particle/f_score‚0.42808952927589417 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 8 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.42740046977996826 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.5254045724868774 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 *jet/accuracy_1_of_3‚0.8660481572151184 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 *jet/accuracy_1_of_4‚0.9114497303962708 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.5416666865348816 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9989210963249207 7 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.3111111521720886 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.18939393758773804 particle/accuracy_3_of_6‚1 & Purity/1FRt1SRqqt3FBt/FRt_purity‚0 :"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.40105175971984863 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.41076385974884033 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚0 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 +jet/accuracy_3_of_6‚0.21153846383094788 7 Purity/1FRt0SRqqt1FBt/FBt_purity‚0.9958823323249817 9"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.4126262366771698 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 , Purity/*FRt2SRqqt2FBt/FRt_purity‚0.34375 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9972519278526306 ;"Purity/2FRt1SRqqt*FBt/event_purity‚0.002721088472753763 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.5018550157546997 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 *jet/accuracy_2_of_4‚0.3594859838485718 ,"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.875 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.31427842378616333 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9989210963249207 ;"Purity/1FRt1SRqqt2FBt/event_purity‚0.010416666977107525 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.04801313951611519 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.01462443359196186 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚0 _runtime‚2839.771108558 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 ) Purity/4FRt1SRqqt*FBt/FRt_purity‚0.25 :"Purity/*FRt1SRqqt2FBt/event_purity‚0.03122120164334774 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9820977449417114 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9969576597213745 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.34062501788139343 particle/accuracy_6_of_6‚1 7 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.3333333432674408 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.03422638773918152 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.5404761433601379 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 particle/accuracy_2_of_6‚1 jet/accuracy_1_of_7‚1 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9899044632911682 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 7 Purity/1FRt4SRqqt*FBt/FRt_purity‚0.3333333432674408 9"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.2916666567325592 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.1907138228416443 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.17309674620628357 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 -"Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.3125 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 5 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.98746258020401 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.5453667044639587 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.9444444179534912 ("Purity/4FRt*SRqqt0FBt/SRqqt_purity‚1 5 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.98064124584198 ,particle/sensitivity‚0.34140709042549133 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9859143495559692 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.16766317188739777 +jet/accuracy_2_of_2‚0.11259179562330246 particle/accuracy_1_of_1‚1 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 ("Purity/0FRt2SRqqt3FBt/SRqqt_purity‚1 _step‚197 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.17691144347190857 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5050210952758789 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.2937605679035187 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 9"Purity/0FRt1SRqqt3FBt/event_purity‚0.1428571492433548 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9793003797531128 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.19124552607536316 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.3194444477558136 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.018694885075092316 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.15944506227970123 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9842017889022827 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.20962537825107574 *jet/accuracy_1_of_1‚0.4902699291706085 particle/accuracy_4_of_5‚1 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6990424394607544 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8094470500946045 6 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.985185444355011 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 particle/accuracy_2_of_5‚1 jet/accuracy_6_of_7‚0 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.2531577944755554 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9970762133598328 ;"Purity/2FRt*SRqqt2FBt/event_purity‚0.029069768264889717 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.4575074315071106 ;"Purity/*FRt3SRqqt0FBt/event_purity‚0.004032257944345474 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 :"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.41333332657814026 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.4545399844646454 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.5184757113456726 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.30000001192092896 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.5309523940086365 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.2916666567325592 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.3214285671710968 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 +jet/accuracy_3_of_4‚0.06181422993540764 particle/accuracy_2_of_4‚1 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.5196823477745056 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 ("Purity/4FRt*SRqqt*FBt/SRqqt_purity‚1 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.5005074739456177 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.22743795812129974 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.16445916891098022 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.5186781883239746 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 :"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.48452115058898926 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5469534397125244 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.07198138535022736 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.4198818802833557 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 :"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.46397724747657776 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.5416666865348816 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9989418387413025 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.1666666716337204 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.46135056018829346 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.4642857015132904 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.13991449773311615 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt3FBt/event_purity‚0.1595744639635086 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9841269850730896 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.0828951969742775 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.08017466217279434 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.5336804986000061 (particle/accuracy‚0.8576182126998901 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9351851940155029 :"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.40555551648139954 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.34375 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5683244466781616 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.5623275637626648 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.5307586193084717 8"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.476775199174881 particle/accuracy_1_of_5‚1 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.30000001192092896 9"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.3499999940395355 jet/accuracy_5_of_7‚0 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5549033880233765 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.5036762356758118 7 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.1764424443244934 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 ( Purity/1FRt*SRqqt4FBt/FRt_purity‚0.5 :"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.48641031980514526 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.2406202107667923 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.3194444477558136 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.4265650510787964 8 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.40416669845581055 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.17083784937858582 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.36526423692703247 <"Purity/1FRt2SRqqt0FBt/event_purity‚0.0014430014416575432 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.15586838126182556 /"Purity/0FRt0SRqqt3FBt/event_purity‚0.203125 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.23223701119422913 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5374752283096313 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.7272727489471436 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚1 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8500000238418579 jet/accuracy_6_of_6‚0 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9952381253242493 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 7validation_average_jet_accuracy‚0.43863746523857117 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 7 Purity/1FRt4SRqqt0FBt/FRt_purity‚0.3333333432674408 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2728697657585144 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9826388955116272 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_5_of_5‚1 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.1111111119389534 +"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.25 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.1795656532049179 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 ;"Purity/0FRt2SRqqt1FBt/event_purity‚0.033018868416547775 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 +"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.75 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9922505021095276 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.6969696879386902 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 @&Purity/4FRt1SRqqt*FBt/event_propor; »ˆTtion‚6.200397183420137e-05 ) Purity/1FRt3SRqqt1FBt/FRt_purity‚0.25 particle/accuracy_5_of_7‚1 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3850546181201935 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 )loss/val_total_loss‚43.19264400685248 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.2517026960849762 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 7 Purity/*FRt*SRqqt4FBt/FBt_purity‚0.8333333134651184 & Purity/*FRt2SRqqt2FBt/FBt_purity‚1 ) Purity/4FRt1SRqqt0FBt/FRt_purity‚0.25 + Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9375 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 7 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.2380952537059784 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 ) Purity/*FRt3SRqqt1FBt/FRt_purity‚0.25 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.5296833515167236 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.8815789222717285 ;"Purity/0FRt3SRqqt0FBt/event_purity‚0.006535947788506746 & Purity/1FRt2SRqqt2FBt/FBt_purity‚1 & Purity/1FRt0SRqqt4FBt/FRt_purity‚1 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.4219032824039459 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.3055555522441864 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.5714285969734192 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚0 :"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.47926953434944153 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 8 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.46317827701568604 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.0555555559694767 ,jet/accuracy_4_of_5‚0.019265232607722282 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5125470757484436 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 particle/accuracy_1_of_3‚1 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.02076777070760727 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.8958333730697632 7 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.2500000298023224 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 ;"Purity/0FRt3SRqqt*FBt/event_purity‚0.006535947788506746 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 particle/accuracy_7_of_7‚0.5 particle/accuracy_1_of_2‚1 particle/accuracy_4_of_4‚1 7 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.8571428656578064 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9883148074150085 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9766082167625427 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 epoch‚1 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9824559688568115 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5181962847709656 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 7 Purity/*FRt3SRqqt1FBt/FBt_purity‚0.9629629850387573 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9816586971282959 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 6 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.132748544216156 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 & Purity/0FRt2SRqqt2FBt/FBt_purity‚1 7 Purity/3FRt1SRqqt2FBt/FRt_purity‚0.1666666716337204 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9867924451828003 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.21801699697971344 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 9"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.4528108537197113 particle/accuracy_2_of_7‚1 + Purity/3FRt3SRqqt0FBt/FRt_purity‚0.1875 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.23857998847961426 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 :"Purity/*FRt0SRqqt2FBt/event_purity‚0.23762166500091553 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 9"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.4136165678501129 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.27142488956451416 8"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.523809552192688 :"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.48672500252723694 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.07202380895614624 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.5055408477783203 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.16255143284797668 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.5438596606254578 7 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.1620796173810959 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.3166666626930237 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 9"Purity/*FRt1SRqqt3FBt/event_purity‚0.1111111119389534 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.18936163187026978 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 +"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.25 9"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.3499999940395355 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.2916666567325592 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.8571428656578064 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 7 Purity/0FRt3SRqqt*FBt/FBt_purity‚0.8888888955116272 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 :"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.40631505846977234 particle/accuracy_1_of_4‚1 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚1 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 *jet/accuracy_1_of_6‚0.9807692170143127 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 +"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.25 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 ;"Purity/*FRt2SRqqt0FBt/event_purity‚0.033172931522130966 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.19907625019550323 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 *jet/accuracy_1_of_5‚0.9475294351577759 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8888888955116272 ("Purity/*FRt2SRqqt3FBt/SRqqt_purity‚1 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9977822303771973 8 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.42901235818862915 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 particle/accuracy_2_of_2‚1 particle/accuracy_1_of_6‚1 jet/accuracy_7_of_7‚0 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.4803331196308136 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.08221767842769623 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_3_of_3‚1 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 particle/accuracy_6_of_7‚1 & Purity/*FRt1SRqqt3FBt/FRt_purity‚0 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.1666666716337204 8"Purity/0FRt0SRqqt2FBt/event_purity‚0.316346138715744 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 +jet/accuracy_3_of_3‚0.02630240097641945 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.11803176254034042 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.25574710965156555 7 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.4431818127632141 8 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.26463964581489563 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 +jet/accuracy_3_of_5‚0.13626471161842346 jet/accuracy_5_of_5‚0 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.1656559258699417 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 *particle/specificity‚0.953150749206543 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 ;"Purity/2FRt0SRqqt2FBt/event_purity‚0.032894738018512726 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 7 Purity/*FRt3SRqqt*FBt/FBt_purity‚0.9642857313156128 7 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.2152777761220932 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9974056482315063 7 Purity/0FRt2SRqqt1FBt/FBt_purity‚0.9811320900917053 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.4270833134651184 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/3FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 :"Purity/1FRt2SRqqt*FBt/event_purity‚0.00186741363722831 *"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.5 particle/accuracy_4_of_6‚1 particle/accuracy_1_of_7‚1 * Purity/4FRt*SRqqt0FBt/FRt_purity‚0.125 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 ;"Purity/1FRt2SRqqt1FBt/event_purity‚0.008620689623057842 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 :"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.32098767161369324 particle/accuracy_3_of_7‚1 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.17971239984035492 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.19528590142726898 8 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.20000000298023224 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5214618444442749 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.2991483807563782 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.4861111044883728 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.1751861572265625 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 7 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.2682328522205353 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.19736842811107635 7 Purity/0FRt3SRqqt1FBt/FBt_purity‚0.8888888955116272 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.23441754281520844 :"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.49342209100723267 jet/accuracy_4_of_7‚0 particle/accuracy_4_of_7‚1 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9792690873146057 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.029735850170254707 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 -"Purity/1FRt0SRqqt3FBt/event_purity‚0.0625 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.029535148292779922 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 & Purity/*FRt3SRqqt2FBt/FRt_purity‚1 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.3656274378299713 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 + Purity/0FRt0SRqqt4FBt/FBt_purity‚0.8125 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 :"Purity/0FRt1SRqqt2FBt/event_purity‚0.05377667769789696 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.36811143159866333 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 *jet/accuracy_1_of_2‚0.7520887851715088 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2380952388048172 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.5099206566810608 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 :"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.37177175283432007 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5303909182548523 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.1003028005361557 :"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.48193812370300293 9"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.2777777910232544 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 $ _timestamp‚1.788513214297978e+09 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9910162687301636 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2730218470096588 7"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.43613401055336 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.2687881290912628 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6488950848579407 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.02443646639585495 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 -jet/accuracy_4_of_4‚0.0040869624353945255 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 + Purity/0FRt*SRqqt4FBt/FBt_purity‚0.8125 jet/accuracy_3_of_7‚0 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.4126685559749603 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9882910847663879 ( Purity/*FRt*SRqqt4FBt/FRt_purity‚0.5 6 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.903073251247406 particle/accuracy_5_of_6‚1 7 Purity/3FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 * Purity/4FRt*SRqqt*FBt/FRt_purity‚0.125 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5125470757484436 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.17777778208255768 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 trainer/global_step‚2417 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 jet/accuracy_2_of_7‚0.5 & Purity/1FRt3SRqqt2FBt/FRt_purity‚1 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9087302684783936 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.17309674620628357 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 *jet/accuracy_2_of_5‚0.4871351718902588 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 8"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.492973655462265 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.26767680048942566 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.3263888955116272 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9680315852165222 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 :"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.41993463039398193 :"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.36666667461395264 +"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.75 & Purity/*FRt0SRqqt4FBt/FRt_purity‚1 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 :"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.43274909257888794 :"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.48996302485466003 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.4908982813358307 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.5044864416122437 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9975355267524719 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.11986728757619858 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.018445847555994987 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5810511708259583 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.010383603163063526 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.4791666865348816 7 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.3529411852359772 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.17273136973381042 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.3819444477558136 jet/accuracy_4_of_6‚0 +"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.25 8 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.31578946113586426 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.17076615989208221 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5242251753807068 :"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.40555551648139954 particle/accuracy_3_of_4‚1 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚0 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.4798871576786041 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.20001950860023499 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.18390804529190063 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.17686063051223755 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.4657277762889862 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.15689656138420105 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.3194444477558136 jet/accuracy_5_of_6‚0 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 + Purity/3FRt3SRqqt*FBt/FRt_purity‚0.1875 ("Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 particle/accuracy_2_of_3‚1 *jet/accuracy_2_of_6‚0.4551282227039337 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 ;"Purity/*FRt3SRqqt*FBt/event_purity‚0.004032257944345474 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.18906398117542267 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.14769743382930756 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9961150288581848 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 particle/accuracy_3_of_5‚1 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.4979395568370819 ;"Purity/2FRt1SRqqt0FBt/event_purity‚0.004497354384511709/ƵzÚ+Àù§ +jet/accuracy_2_of_2‚0.11259179562330246 particle/accuracy_1_of_1‚1 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 ("Purity/0FRt2SRqqt3FBt/SRqqt_purity‚1 _step‚197 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.17691144347190857 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5050210952758789 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.2937605679035187 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 9"Purity/0FRt1SRqqt3FBt/event_purity‚0.1428571492433548 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9793003797531128 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.19124552607536316 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.3194444477558136 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.018694885075092316 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.15944506227970123 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9842017889022827 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.20962537825107574 *jet/accuracy_1_of_1‚0.4902699291706085 particle/accuracy_4_of_5‚1 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6990424394607544 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8094470500946045 6 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.985185444355011 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 particle/accuracy_2_of_5‚1 jet/accuracy_6_of_7‚0 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.2531577944755554 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9970762133598328 ;"Purity/2FRt*SRqqt2FBt/event_purity‚0.029069768264889717 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.4575074315071106 ;"Purity/*FRt3SRqqt0FBt/event_purity‚0.004032257944345474 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 :"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.41333332657814026 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.4545399844646454 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.5184757113456726 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.30000001192092896 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.5309523940086365 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.2916666567325592 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.3214285671710968 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 +jet/accuracy_3_of_4‚0.06181422993540764 particle/accuracy_2_of_4‚1 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.5196823477745056 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 ("Purity/4FRt*SRqqt*FBt/SRqqt_purity‚1 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.5005074739456177 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.22743795812129974 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.16445916891098022 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.5186781883239746 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 :"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.48452115058898926 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5469534397125244 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.07198138535022736 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.4198818802833557 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 :"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.46397724747657776 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.5416666865348816 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9989418387413025 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.1666666716337204 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.46135056018829346 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.4642857015132904 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.13991449773311615 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt3FBt/event_purity‚0.1595744639635086 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9841269850730896 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.0828951969742775 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.08017466217279434 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.5336804986000061 (particle/accuracy‚0.8576182126998901 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9351851940155029 :"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.40555551648139954 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.34375 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5683244466781616 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.5623275637626648 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.5307586193084717 8"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.476775199174881 particle/accuracy_1_of_5‚1 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.30000001192092896 9"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.3499999940395355 jet/accuracy_5_of_7‚0 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5549033880233765 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.5036762356758118 7 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.1764424443244934 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 ( Purity/1FRt*SRqqt4FBt/FRt_purity‚0.5 :"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.48641031980514526 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.2406202107667923 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.3194444477558136 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.4265650510787964 8 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.40416669845581055 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.17083784937858582 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.36526423692703247 <"Purity/1FRt2SRqqt0FBt/event_purity‚0.0014430014416575432 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.15586838126182556 /"Purity/0FRt0SRqqt3FBt/event_purity‚0.203125 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.23223701119422913 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5374752283096313 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.7272727489471436 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚1 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8500000238418579 jet/accuracy_6_of_6‚0 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9952381253242493 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 7validation_average_jet_accuracy‚0.43863746523857117 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 7 Purity/1FRt4SRqqt0FBt/FRt_purity‚0.3333333432674408 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2728697657585144 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9826388955116272 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_5_of_5‚1 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.1111111119389534 +"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.25 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.1795656532049179 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 ;"Purity/0FRt2SRqqt1FBt/event_purity‚0.033018868416547775 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 +"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.75 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9922505021095276 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.6969696879386902 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 ) Purity/1FRt3SRqqt1FBt/FRt_purity‚0.25 particle/accuracy_5_of_7‚1 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3850546181201935 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 )loss/val_total_loss‚43.19264400685248 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.2517026960849762 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 7 Purity/*FRt*SRqqt4FBt/FBt_purity‚0.8333333134651184 & Purity/*FRt2SRqqt2FBt/FBt_purity‚1 ) Purity/4FRt1SRqqt0FBt/FRt_purity‚0.25 + Purity/1FRt0SRqqt3FBt/FBt_purity‚%Àv&h0.9375 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 7 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.2380952537059784 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 ) Purity/*FRt3SRqqt1FBt/FRt_purity‚0.25 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.5296833515167236 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.8815789222717285 ;"Purity/0FRt3SRqqt0FBt/event_purity‚0.006535947788506746 & Purity/1FRt2SRqqt2FBt/FBt_purity‚1 & Purity/1FRt0SRqqt4FBt/FRt_purity‚1 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.4219032824039459 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.3055555522441864 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.5714285969734192 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚0 :"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.47926953434944153 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 8 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.46317827701568604 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.0555555559694767 ,jet/accuracy_4_of_5‚0.019265232607722282 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5125470757484436 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 particle/accuracy_1_of_3‚1 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.02076777070760727 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.8958333730697632 7 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.2500000298023224 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 ;"Purity/0FRt3SRqqt*FBt/event_purity‚0.006535947788506746 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 particle/accuracy_7_of_7‚0.5 particle/accuracy_1_of_2‚1 particle/accuracy_4_of_4‚1 7 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.8571428656578064 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9883148074150085 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9766082167625427 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 epoch‚1 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9824559688568115 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5181962847709656 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 7 Purity/*FRt3SRqqt1FBt/FBt_purity‚0.9629629850387573 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9816586971282959 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 6 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.132748544216156 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 & Purity/0FRt2SRqqt2FBt/FBt_purity‚1 7 Purity/3FRt1SRqqt2FBt/FRt_purity‚0.1666666716337204 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9867924451828003 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.21801699697971344 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 9"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.4528108537197113 particle/accuracy_2_of_7‚1 + Purity/3FRt3SRqqt0FBt/FRt_purity‚0.1875 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.23857998847961426 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 :"Purity/*FRt0SRqqt2FBt/event_purity‚0.23762166500091553 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 9"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.4136165678501129 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.27142488956451416 8"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.523809552192688 :"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.48672500252723694 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.07202380895614624 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.5055408477783203 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.16255143284797668 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.5438596606254578 7 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.1620796173810959 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.3166666626930237 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 9"Purity/*FRt1SRqqt3FBt/event_purity‚0.1111111119389534 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.18936163187026978 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 +"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.25 9"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.3499999940395355 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.2916666567325592 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.8571428656578064 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 7 Purity/0FRt3SRqqt*FBt/FBt_purity‚0.8888888955116272 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 :"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.40631505846977234 particle/accuracy_1_of_4‚1 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚1 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 *jet/accuracy_1_of_6‚0.9807692170143127 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 +"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.25 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 ;"Purity/*FRt2SRqqt0FBt/event_purity‚0.033172931522130966 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.19907625019550323 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 *jet/accuracy_1_of_5‚0.9475294351577759 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8888888955116272 ("Purity/*FRt2SRqqt3FBt/SRqqt_purity‚1 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9977822303771973 8 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.42901235818862915 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 particle/accuracy_2_of_2‚1 particle/accuracy_1_of_6‚1 jet/accuracy_7_of_7‚0 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.4803331196308136 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.08221767842769623 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_3_of_3‚1 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 particle/accuracy_6_of_7‚1 & Purity/*FRt1SRqqt3FBt/FRt_purity‚0 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.1666666716337204 8"Purity/0FRt0SRqqt2FBt/event_purity‚0.316346138715744 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 +jet/accuracy_3_of_3‚0.02630240097641945 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.11803176254034042 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.25574710965156555 7 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.4431818127632141 8 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.26463964581489563 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 +jet/accuracy_3_of_5‚0.13626471161842346 jet/accuracy_5_of_5‚0 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.1656559258699417 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 *particle/specificity‚0.953150749206543 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 ;"Purity/2FRt0SRqqt2FBt/event_purity‚0.032894738018512726 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 7 Purity/*FRt3SRqqt*FBt/FBt_purity‚0.9642857313156128 7 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.2152777761220932 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9974056482315063 7 Purity/0FRt2SRqqt1FBt/FBt_purity‚0.9811320900917053 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.4270833134651184 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/3FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 :"Purity/1FRt2SRqqt*FBt/event_purity‚0.00186741363722831 *"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.5 particle/accuracy_4_of_6‚1 particle/accuracy_1_of_7‚1 * Purity/4FRt*SRqqt0FBt/FRt_purity‚0.125 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 ;"Purity/1FRt2SRqqt1FBt/event_purity‚0.008620689623057842 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 :"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.32098767161369324 particle/accuracy_3_of_7‚1 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.17971239984035492 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.19528590142726898 8 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.20000000298023224 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5214618444442749 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.2991483807563782 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.4861111044883728 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.1751861572265625 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 7 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.2682328522205353 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.19736842811107635 7 Purity/0FRt3SRqqt1FBt/FBt_purity‚0.8888888955116272 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.23441754281520844 :"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.49342209100723267 jet/accuracy_4_of_7‚0 particle/accuracy_4_of_7‚1 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9792690873146057 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.029735850170254707 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 -"Purity/1FRt0SRqqt3FBt/event_purity‚0.0625 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.029535148292779922 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 & Purity/*FRt3SRqqt2FBt/FRt_purity‚1 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.3656274378299713 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 + Purity/0FRt0SRqqt4FBt/FBt_purity‚0.8125 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 :"Purity/0FRt1SRqqt2FBt/event_purity‚0.05377667769789696 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.36811143159866333 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 *jet/accuracy_1_of_2‚0.7520887851715088 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2380952388048172 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.5099206566810608 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 :"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.37177175283432007 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5303909182548523 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.1003028005361557 :"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.48193812370300293 9"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.2777777910232544 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 $ _timestamp‚1.788513214297978e+09 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9910162687301636 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2730218470096588 7"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.43613401055336 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.2687881290912628 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6488950848579407 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.02443646639585495 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 -jet/accuracy_4_of_4‚0.0040869624353945255 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 + Purity/0FRt*SRqqt4FBt/FBt_purity‚0.8125 jet/accuracy_3_of_7‚0 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.4126685559749603 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9882910847663879 ( Purity/*FRt*SRqqt4FBt/FRt_purity‚0.5 6 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.903073251247406 particle/accuracy_5_of_6‚1 7 Purity/3FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 * Purity/4FRt*SRqqt*FBt/FRt_purity‚0.125 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5125470757484436 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.17777778208255768 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 trainer/global_step‚2417 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 jet/accuracy_2_of_7‚0.5 & Purity/1FRt3SRqqt2FBt/FRt_purity‚1 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9087302684783936 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.17309674620628357 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 *jet/accuracy_2_of_5‚0.4871351718902588 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 8"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.492973655462265 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.26767680048942566 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.3263888955116272 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9680315852165222 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 :"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.41993463039398193 :"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.36666667461395264 +"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.75 & Purity/*FRt0SRqqt4FBt/FRt_purity‚1 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 :"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.43274909257888794 :"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.48996302485466003 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.4908982813358307 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.5044864416122437 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9975355267524719 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.11986728757619858 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.018445847555994987 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5810511708259583 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.010383603163063526 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.4791666865348816 7 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.3529411852359772 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.17273136973381042 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.3819444477558136 jet/accuracy_4_of_6‚0 +"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.25 8 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.31578946113586426 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.17076615989208221 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5242251753807068 :"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.40555551648139954 particle/accuracy_3_of_4‚1 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚0 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.4798871576786041 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.20001950860023499 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.18390804529190063 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.17686063051223755 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.4657277762889862 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.15689656138420105 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.3194444477558136 jet/accuracy_5_of_6‚0 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 + Purity/3FRt3SRqqt*FBt/FRt_purity‚0.1875 ("Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 particle/accuracy_2_of_3‚1 *jet/accuracy_2_of_6‚0.4551282227039337 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 ;"Purity/*FRt3SRqqt*FBt/event_purity‚0.004032257944345474 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.18906398117542267 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.14769743382930756 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9961150288581848 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 particle/accuracy_3_of_5‚1 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.4979395568370819 ;"Purity/2FRt1SRqqt0FBt/event_purity‚0.004497354384511709 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 +jet/accuracy_2_of_3‚0.24936501681804657 8"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.949999988079071 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 :"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.49845677614212036 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 (particle/f_score‚0.42808952927589417 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 8 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.42740046977996826 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.5254045724868774 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 *jet/accuracy_1_of_3‚0.8660481572151184 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 *jet/accuracy_1_of_4‚0.9114497303962708 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.5416666865348816 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9989210963249207 7 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.3111111521720886 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.18939393758773804 particle/accuracy_3_of_6‚1 & Purity/1FRt1SRqqt3FBt/FRt_purity‚0 :"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.40105175971984863 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.41076385974884033 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚0 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 +jet/accuracy_3_of_6‚0.21153846383094788 7 Purity/1FRt0SRqqt1FBt/FBt_purity‚0.9958823323249817 9"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.4126262366771698 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 , Purity/*FRt2SRqqt2FBt/FRt_purity‚0.34375 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9972519278526306 ;"Purity/2FRt1SRqqt*FBt/event_purity‚0.002721088472753763 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.5018550157546997 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 *jet/accuracy_2_of_4‚0.3594859838485718 ,"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.875 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.31427842378616333 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9989210963249207 ;"Purity/1FRt1SRqqt2FBt/event_purity‚0.010416666977107525 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.04801313951611519 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.01462443359196186 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚0 _runtime‚2839.771108558 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 ) Purity/4FRt1SRqqt*FBt/FRt_purity‚0.25 :"Purity/*FRt1SRqqt2FBt/event_purity‚0.03122120164334774 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9820977449417114 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9969576597213745 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.34062501788139343 particle/accuracy_6_of_6‚1 7 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.3333333432674408 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.03422638773918152 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.5404761433601379 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 particle/accuracy_2_of_6‚1 jet/accuracy_1_of_7‚1 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9899044632911682 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 7 Purity/1FRt4SRqqt*FBt/FRt_purity‚0.3333333432674408 9"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.2916666567325592 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.1907138228416443 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.17309674620628357 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 -"Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.3125 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 5 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.98746258020401 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.5453667044639587 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.9444444179534912 ("Purity/4FRt*SRqqt0FBt/SRqqt_purity‚1 5 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.98064124584198 ,particle/sensitivity‚0.34140709042549133 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9859143495559692 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.16766317188739777Å‚ƒqå”Á‚(:‰ È—êÔ”›Çý gpu.0.gpu‚47.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚36.0 gpu.0.powerWatts‚119.091' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚29.77275 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ó*ƒˆd‚(:Ù È—êÔ³¢³‚ network.sent‚ 5.175873e+06 disk.sda1.in‚0 disk.nvme0n1.in‚ 28.52734375 disk.nvme1n1.in‚ 184.96484375) proc.memory.availableMB‚ 1301.05078125 disk.sdb2.in‚0.03125" disk.nvme0n1.out‚ 16824.0546875 disk.rbd0.in‚0 proc.cpu.threads‚17! disk.sdb2.out‚325.28271484375 disk.dm-0.out‚ 31373.0859375 disk.md0p1.in‚ 0.0703125! disk.sda2.out‚325.28271484375 disk.rbd0.out‚ 0.845703125 cpu‚25.23555176913671 network.recv‚946255 disk.md0.in‚ 0.0703125 disk.sda1.out‚0 disk.sda.out‚325.28271484375 disk.sdb1.in‚0) disk./.usagePercent‚75.81454184523062 disk.md0.out‚ 313.27734375 disk.sda2.in‚ 0.0390625 disk.sdb1.out‚0) proc.memory.percent‚18.31996440887451 disk.dm-0.in‚ 213.4921875 disk.md0p1.out‚ 313.27734375 disk.sdb.in‚0.03125 disk.sda.in‚ 0.0390625$ memory_percent‚92.05901622772217 disk.rbd2.out‚ 0.783203125 disk.sdb.out‚325.28271484375 disk.nvme1n1.out‚ 14549.03125% disk./.usageGB‚2711.4952430725098 disk.rbd2.in‚0# proc.memory.rssMB‚ 3001.54296875|_F‚VÂ(:Ë ×—êÔ¾³¸ÿ disk.md0p1.out‚ 315.1484375 disk.sda2.in‚ 0.0390625 disk.sdb1.in‚0 disk.sda1.out‚0 disk.dm-0.out‚ 31458.703125 network.recv‚951098 disk.md0.out‚ 315.1484375 disk.sda.in‚ 0.0390625 disk.sdb1.out‚0 disk.sdb.out‚ 327.2109375 disk.rbd2.out‚ 0.783203125 disk.sdb2.in‚0.03125 disk.sda1.in‚0) proc.memory.percent‚18.31996440887451 proc.cpu.threads‚17 disk.nvme1n1.in‚ 184.96484375 network.sent‚ 5.288043e+06% disk./.usageGB‚2711.3585090637207 disk.sda.out‚ 327.2109375 disk.rbd0.out‚ 0.845703125 disk.sdb2.out‚ 327.2109375 disk.nvme0n1.in‚ 28.52734375) proc.memory.availableMB‚ 1301.41015625 disk.sdb.in‚0.03125 disk.rbd0.in‚0 disk.md0.in‚ 0.0703125 disk.dm-0.in‚ 213.4921875# proc.memory.rssMB‚ 3001.54296875 cpu‚25.234434665332486) disk./.usagePercent‚75.81071870511725 disk.sda2.out‚ 327.2109375 disk.md0p1.in‚ 0.0703125$ memory_percent‚92.05682277679443# disk.nvme1n1.out‚14633.83203125 disk.rbd2.in‚0# disk.nvme0n1.out‚16824.87109375ýĨܒÄ‚(:‡ ×—êÔ—·åÿ gpu.0.gpu‚42.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚163.09' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚40.7725 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0dŽ‡2Å- _wandbruntime‚2878  _runtime‚2878‰š¿ÇÆÁ % _timestamp‚1.7885132531808968e+09 _runtime‚2878.651464999 _step‚198 & lr-AdamW/pg1‚0.0008305088339332231 & lr-AdamW/pg2‚0.0008305088339332231 trainer/global_step‚2449fàc[ÌÇÆ & lr-AdamW/pg1‚0.0008305088339332231 & lr-AdamW/pg2‚0.0008305088339332231 trainer/global_step‚2449 % _timestamp‚1.7885132531808968e+09 _runtime‚2878.651464999 _step‚198ÆæzNù2È- _wandbruntime‚2878  _runtime‚2878߯÷½ ÉìO PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚50685532 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407360 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚228749606060032 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚46311231 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450776 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚61537398 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2788309133824 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚61511311 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚606 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚604 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚50685530 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26089 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 % _timestamp‚1.7885132531819787e+09 _runtime‚2878.652519244 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2778812165023 _step‚199 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31685 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚204658835235328 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 112222930 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2777914753607 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31930 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26087 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2788309133824 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚14794538 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚204657917825024 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚201869626694144 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31930 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26087 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚197343822016944 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410304 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚201869608691200 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2910817670656 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚31516695 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚201869608691200 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2789208541184 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177046528 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1180 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26089 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411416 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1205 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 trainer/global_step‚2449 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚228749865152000 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917410304 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚14795142 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31685 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚204658835235328 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2789208541184 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177046528 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚204657917825024 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚61537398 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚197343839056304 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 112196841 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚200122651221327 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚50685532 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2910582518784 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955590656 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚201869626694144 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 398533632 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚61511311 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚225839047481344 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚225839023541248 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 235151872ÂÕhÊ÷'ÊñO G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚31516695 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚201869608691200 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2789208541184 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177046528 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1180 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26089 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411416 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1205 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 trainer/global_step‚2449 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚228749865152000 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917410304 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚14795142 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31685 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚204658835235328 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2789208541184 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177046528 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚204657917825024 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚61537398 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚197343839056304 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 112196841 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚200122651221327 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚50685532 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2910582518784 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955590656 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚201869626694144 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 398533632 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚61511311 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚225839047481344 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚225839023541248 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 235151872 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚50685532 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407360 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚228749606060032 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚46311231 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450776 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚61537398 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2788309133824 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚61511311 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚606 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚604 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚50685530 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26089 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 % _timestamp‚1.7885132531819787e+09 _runtime‚2878.652519244 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2778812165023 _step‚199 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31685 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚204658835235328 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 112222930 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2777914753607 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31930 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26087 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2788309133824 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚14794538 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚204657917825024 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚201869626694144 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31930 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26087 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚197343822016944 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410304 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚201869608691200 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2910817670656 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984ÇB·¡2Ë- _wandbruntime‚2879  _runtime‚2879¼åpÝû&ÌõM =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚46329692 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31930 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26113 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2789421937152 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917423616 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚50705428 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2790321357824 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚228840088508928 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 398533632 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31930 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚50705428 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚201949256989696 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚609 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚228840335004672 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 246495744 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214661 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 112267445 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚61535902 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚31529071 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚200201739976981 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚225928366702080 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917423616 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899420672 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚14800621 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚46329081 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14800012 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚225928342761984 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 112241330 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 222555648 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚31529069 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2911745746944 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411624 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚611 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 112241330 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 trainer/global_step‚2449 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26115 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450984 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2911968302592 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899420672 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚197421819150432 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 _step‚200 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚197421802111072 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955590656 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚204739578347520 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2779023414925 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚50705430 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚201949238986752 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177046528 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 üNJ„ÍúM [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚201949256989696 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚609 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚228840335004672 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 246495744 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214661 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 112267445 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚61535902 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚31529071 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚200201739976981 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚225928366702080 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917423616 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899420672 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚14800621 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚46329081 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14800012 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_poolÄ¡º|.freed‚225928342761984 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 112241330 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 222555648 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚31529069 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2911745746944 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411624 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚611 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 112241330 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 trainer/global_step‚2449 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26115 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450984 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2911968302592 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899420672 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚197421819150432 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 _step‚200 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚197421802111072 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955590656 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚204739578347520 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2779023414925 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚50705430 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚201949238986752 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177046528 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚46329692 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31930 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26113 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2789421937152 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917423616 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚50705428 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2790321357824 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚228840088508928 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 398533632 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31930 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚50705428 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728ÈÞŸ~2Î- _wandbruntime‚2879  _runtime‚2879Ë´/SÏÍ 0loss/FRt4/assignment_loss‚3.3839852724756514 1loss/FBt4/assignment_loss‚0.22262037800499002 .loss/FRt3/detection_loss‚2.482059161079691 0loss/FRt2/assignment_loss‚1.3558576741944188 /loss/FBt2/detection_loss‚0.7641792036838045 $loss/total_loss‚41.8416228988318 epoch‚2 trainer/global_step‚2449 0loss/FRt1/assignment_loss‚3.3662639613054237 2loss/SRqqt1/assignment_loss‚1.8304720226189364 .loss/FRt1/detection_loss‚2.041806019690573 -loss/FRt4/detection_loss‚2.68032044172287 .loss/FBt3/detection_loss‚2.422572680882045 2loss/SRqqt2/assignment_loss‚0.8887493836298359 1loss/SRqqt3/assignment_loss‚1.589608601921675 $ _timestamp‚1.788513254404125e+09 0loss/FRt3/assignment_loss‚2.4262128649873937 0loss/FBt3/assignment_loss‚0.3061820217946466 1loss/SRqqt1/detection_loss‚1.8036950437918953 1loss/SRqqt2/detection_loss‚0.9523729884037466 _runtime‚2879.874283376 1loss/SRqqt4/assignment_loss‚2.046761433993067 0loss/FBt1/assignment_loss‚0.1734443255977567 .loss/FBt2/assignment_loss‚0.13458067204415 1loss/SRqqt3/detection_loss‚1.9707690749545486 /loss/FBt1/detection_loss‚2.7962865978479385 /loss/FRt2/detection_loss‚1.7966092421632747 1loss/SRqqt4/detection_loss‚2.0335739027886164 -loss/FBt4/detection_loss‚2.37263992925485 _step‚201 MbkXÐÒ 0loss/FRt2/assignment_loss‚1.3558576741944188 /loss/FBt2/detection_loss‚0.7641792036838045 $loss/total_loss‚41.8416228988318 epoch‚2 trainer/global_step‚2449 0loss/FRt1/assignment_loss‚3.3662639613054237 2loss/SRqqt1/assignment_loss‚1.8304720226189364 .loss/FRt1/detection_loss‚2.041806019690573 -loss/FRt4/detection_loss‚2.68032044172287 .loss/FBt3/detection_loss‚2.422572680882045 2loss/SRqqt2/assignment_loss‚0.8887493836298359 1loss/SRqqt3/assignment_loss‚1.589608601921675 $ _timestamp‚1.788513254404125e+09 0loss/FRt3/assignment_loss‚2.4262128649873937 0loss/FBt3/assignment_loss‚0.3061820217946466 1loss/SRqqt1/detection_loss‚1.8036950437918953 1loss/SRqqt2/detection_loss‚0.9523729884037466 _runtime‚2879.874283376 1loss/SRqqt4/assignment_loss‚2.046761433993067 0loss/FBt1/assignment_loss‚0.1734443255977567 .loss/FBt2/assignment_loss‚0.13458067204415 1loss/SRqqt3/detection_loss‚1.9707690749545486 /loss/FBt1/detection_loss‚2.7962865978479385 /loss/FRt2/detection_loss‚1.7966092421632747 1loss/SRqqt4/detection_loss‚2.0335739027886164 -loss/FBt4/detection_loss‚2.37263992925485 _step‚201 0loss/FRt4/assignment_loss‚3.3839852724756514 1loss/FBt4/assignment_loss‚0.22262037800499002 .loss/FRt3/detection_loss‚2.482059161079691É™ù›Ø“Ñ‚(:ˆ æ—êÔå¹Þý gpu.0.gpu‚2.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚37.0 gpu.0.powerWatts‚170.891' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.72275 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0«þ¾ñ^Ò‚(:Ó æ—êÔߥ€# memory_percent‚92.0628547668457 disk.sdb1.out‚0 disk.md0.out‚ 316.55859375 disk.sda1.in‚0 cpu‚25.23338407846317$ disk./.usageGB‚2711.572048187256 disk.sda.out‚328.67041015625 disk.sda2.in‚ 0.0390625 disk.sda1.out‚0 disk.md0p1.in‚ 0.0703125 disk.sda.in‚ 0.0390625 disk.rbd0.out‚ 0.845703125! disk.sdb2.out‚328.67041015625' proc.memory.availableMB‚ 1300.421875! proc.memory.rssMB‚ 3001.546875 proc.cpu.threads‚17 disk.sdb.out‚328.67041015625# disk.nvme1n1.out‚14711.14453125 disk.rbd2.in‚0 disk.sdb2.in‚0.03125 disk.nvme0n1.out‚17544.25 network.sent‚ 5.304961e+06 disk.md0.in‚ 0.0703125 disk.rbd2.out‚ 0.783203125 disk.md0p1.out‚ 316.55859375* proc.memory.percent‚18.319988250732422) disk./.usagePercent‚75.81668934838436 disk.sdb.in‚0.03125 disk.nvme1n1.in‚ 184.96484375 network.recv‚955739! disk.sda2.out‚328.67041015625 disk.rbd0.in‚0 disk.dm-0.out‚32255.39453125 disk.nvme0n1.in‚ 28.52734375 disk.sdb1.in‚0 disk.dm-0.in‚ 213.4921875”Š(¼“Ó‚(:ˆ õ—êÔÒø¤ÿ gpu.0.gpu‚49.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚165.266' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚41.3165 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0þÓ¡t_Ô‚(:Ô õ—êÔŸÿÆ€ disk.rbd2.in‚0 disk.sdb2.in‚0.03125 disk.sda1.in‚0 disk.md0p1.in‚ 0.0703125 disk.sdb.in‚0.03125 disk.nvme1n1.in‚ 184.96484375 disk.sda1.out‚0 disk.nvme0n1.in‚ 28.52734375 disk.sda2.in‚ 0.0390625 disk.rbd0.in‚0 disk.rbd0.out‚ 0.845703125* proc.memory.percent‚18.319988250732422 disk.dm-0.in‚ 213.4921875 disk.sda.in‚ 0.0390625 disk.sdb1.in‚0 disk.md0.out‚ 318.109375 disk.rbd2.out‚ 0.783203125! disk.sdb2.out‚330.28564453125 disk.sda.out‚330.28564453125! disk.nvme1n1.out‚ 14742.703125 disk.sdb1.out‚0 network.sent‚ 5.401248e+06# disk.nvme0n1.out‚17615.32421875 disk.sdb.out‚330.28564453125! disk.sda2.out‚330.28564453125$ memory_percent‚92.05646514892578! proc.memory.rssMB‚ 3001.546875) disk./.usagePercent‚75.81854641490912% disk./.usageGB‚2711.6384658813477 disk.md0.in‚ 0.0703125 cpu‚25.23226322802337 proc.cpu.threads‚17 network.recv‚960347 disk.md0p1.out‚ 318.109375 disk.dm-0.out‚32358.02734375& proc.memory.availableMB‚ 1301.46875Å¿bOžÕ‚(:“ „˜êÔ«çíþ gpu.0.gpu‚42.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚37.0 gpu.0.powerWatts‚127.106' gpu.0.enforcedPowerLimitWatts‚400.0) gpu.0.powerPercent‚31.776499999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚08B­Ð`Ö‚(:Õ „˜êÔÊ÷›ÿ disk.sda.in‚ 0.0390625! disk.nvme1n1.out‚ 14887.203125 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.md0.out‚ 320.1484375 disk.sda1.in‚0 disk.md0p1.in‚ 0.0703125 disk.md0p1.out‚ 320.1484375 disk.sdb.in‚0.03125 network.recv‚964840$ memory_percent‚92.05856323242188$ proc.memory.availableMB‚1301.125) disk./.usagePercent‚75.82552883998395 disk.rbd2.in‚0 disk.sdb2.in‚0.03125! disk.sdb2.out‚332.38623046875 disk.sda1.out‚0 cpu‚25.231185471732765 disk.rbd0.out‚ 0.845703125 disk.nvme1n1.in‚ 184.96484375 disk.rbd2.out‚ 0.783203125 disk.sda.out‚332.38623046875! disk.sda2.out‚332.38623046875 disk.md0.in‚ 0.0703125# disk.nvme0n1.out‚17669.57421875* proc.memory.percent‚18.319988250732422 proc.cpu.threads‚17 disk.sda2.in‚ 0.0390625 disk.dm-0.out‚32556.77734375 disk.sdb.out‚332.38623046875! proc.memory.rssMB‚ 3001.546875 disk.nvme0n1.in‚ 28.52734375 network.sent‚ 5.407209e+06 disk.rbd0.in‚0 disk.dm-0.in‚ 213.4921875% disk./.usageGB‚2711.88819122314459«9Eׂ(:… “˜êÔ“þ gpu.0.gpu‚49.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚161.7' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚40.425 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0A?‚þ^Ø‚(:Ó “˜êÔØŽýÿ disk.sdb1.in‚0 cpu‚25.23016622252625 disk.rbd2.in‚0 disk.sdb2.in‚0.03125 disk.nvme0n1.in‚ 28.52734375 disk.sdb.out‚333.89306640625 disk.rbd0.in‚0 proc.cpu.threads‚17 disk.sda.out‚333.89306640625 disk.rbd0.out‚ 0.845703125 disk.md0.out‚ 321.59375 disk.dm-0.in‚ 213.4921875 disk.dm-0.out‚32557.49609375# disk.nvme1n1.out‚14887.55078125! proc.memory.rssMB‚ 3001.546875* proc.memory.percent‚18.319988250732422! disk.sdb2.out‚333.89306640625" disk.nvme0n1.out‚ 17669.9453125 disk.sda.in‚ 0.0390625! disk.sda2.out‚333.89306640625 disk.sda1.in‚0 disk.sda1.out‚0 network.sent‚ 5.41308e+06 disk.md0p1.in‚ 0.0703125 disk.sda2.in‚ 0.0390625 disk.md0.in‚ 0.0703125 disk.rbd2.out‚ 0.783203125 disk.nvme1n1.in‚ 184.96484375 disk.sdb.in‚0.03125 network.recv‚968836) disk./.usagePercent‚75.83781293346841% disk./.usageGB‚2712.3275299072266 disk.sdb1.out‚0 disk.md0p1.out‚ 321.59375$ memory_percent‚92.05999374389648' proc.memory.availableMB‚ 1300.890625  –p2Ù- _wandbruntime‚2936  _runtime‚2936@ÁbÆÚÀ _runtime‚ 2936.18519354 _step‚202 & lr-AdamW/pg1‚0.0008282976454269042 & lr-AdamW/pg2‚0.0008282976454269042 trainer/global_step‚2499 % _timestamp‚1.7885133107147286e+09¤Z´ôËÛÅ _step‚202 & lr-AdamW/pg1‚0.0008282976454269042 & lr-AdamW/pg2‚0.0008282976454269042 trainer/global_step‚2499 % _timestamp‚1.7885133107147286e+09 _runtime‚ 2936.18519354Ê<¹=¯2Ü- _wandbruntime‚2936  _runtime‚2936,`K·ò'ÝìO XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407360 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚204077102584835 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26089 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26087 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2834258822131 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚62770300 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 114450732 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚47212523 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚605 trainer/global_step‚2499 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407360 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚51680432 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177046528 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚62770300 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚233273651946496 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚201242826723344 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚208695087182848 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2833361410715 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1205 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 114424643 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2843963714048 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚204076188134059 _runtime‚2936.186566747 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917410304 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2844863121408 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450776 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚62744213 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚51680430 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚230305008518144 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2968643428352 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31685 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31685 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚51680432 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214661 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚205851141471744 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31930 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 233054720 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _step‚203 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚208695087182848 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26087 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚51680430 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚62744213 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚233273394951680 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007237 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 256994816 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚230304984578048 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955590656 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2968410373632 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚208696004593152 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚36779¹€sÞñO eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚230305008518144 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2968643428352 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31685 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31685 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚51680432 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214661 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚205851141471744 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31930 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 233054720 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _step‚203 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚208695087182848 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26087 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚51680430 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚62744213 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚233273394951680 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007237 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 256994816 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚230304984578048 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955590656 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2968410373632 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚208696004593152 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407360 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚204077102584835 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26089 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26087 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2834258822131 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚62770300 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 114450732 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚47212523 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚605 trainer/global_step‚2499 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407360 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚51680432 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177046528 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚62770300 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚233273651946496 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚201242826723344 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚208695087182848 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2833361410715 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1205 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F8DeviceStatsMonitor.on_train_batúÿ„ ch_start/active.all.freed‚ 114424643 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2843963714048 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚204076188134059 _runtime‚2936.186566747 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917410304 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2844863121408 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450776 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚62744213 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚51680430˘ÅÄ÷2ß- _wandbruntime‚2937  _runtime‚2937'Šü&àöM QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 398533632 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2969563544064 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚51700328 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚62768804 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚15082048 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2845076517376 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚205930771767296 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899420672 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚208776747705344 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚233363867342848 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚230394303798784 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450984 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 114469132 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚51700328 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214661 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411624 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26115 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 114495247 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 256981504 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚230394327738880 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917423616 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177046528 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 233041408 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚51700330 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1205 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 % _timestamp‚1.7885133117571578e+09 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2845975938048 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚205930753764352 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚201320806817472 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚205930771767296 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 trainer/global_step‚2499 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚47229917 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚32147869 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955590656 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26113 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚208776747705344 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177046528 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚15082650 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚62794917 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚62768804 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2969796585472 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2845975938048 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚47230521 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 114495247 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚62794917 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 Aµ³'áûM SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899420672 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚208776747705344 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚233363867342848 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚230394303798784 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450984 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 114469132 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚51700328 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214661 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411624 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26115 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 114495247 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 256981504 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚230394327738880 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917423616 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177046528 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 233041408 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚51700330 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1205 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 % _timestamp‚1.7885133117571578e+09 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2845975938048 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚205930753764352 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚201320806817472 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚205930771767296 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 trainer/global_step‚2499 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚47229917 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚32147869 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955590656 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26113 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚208776747705344 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177046528 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚15082650 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚62794917 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚62768804 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2969796585472 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2845975938048 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚47230521 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 114495247 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚62794917 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 398533632 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2969563544064 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚51700328 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚62768804 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚15082048 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2845076517376 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚205930771767296ÌÜS9û2â- _wandbruntime‚2937  _runtime‚2937J½ìXXãÒ 0loss/FRt3/assignment_loss‚3.2413673043251037 /loss/FBt3/detection_loss‚2.9105795877320424 _runtime‚2937.229431338 1loss/SRqqt1/assignment_loss‚1.727653979605367 /loss/FRt2/detection_loss‚1.4097465963340272 .loss/FRt3/detection_loss‚2.573567580514484 2loss/SRqqt2/assignment_loss‚1.0116924549846982 1loss/SRqqt3/assignment_loss‚1.114965204579326 1loss/FBt1/assignment_loss‚0.22137576739255566 1loss/FBt3/assignment_loss‚0.33068018885595457 0loss/SRqqt4/detection_loss‚2.761251655717691 %loss/total_loss‚43.82246839184083 % _timestamp‚1.7885133117592833e+09 /loss/FRt4/assignment_loss‚3.304212176281473 /loss/FRt1/detection_loss‚2.4049412007087336 0loss/SRqqt1/detection_loss‚2.155673865662065 1loss/SRqqt3/detection_loss‚1.7822335863915773 .loss/FBt1/detection_loss‚2.652783566713333 epoch‚2 0loss/FRt1/assignment_loss‚3.9158460358355907 0loss/FRt2/assignment_loss‚1.5558652457068949 1loss/SRqqt2/detection_loss‚0.8964392006015166 /loss/FBt4/detection_loss‚1.9700902104377747 1loss/FBt2/assignment_loss‚0.11356448443315718 1loss/FBt4/assignment_loss‚0.39544105477375524 .loss/FRt4/detection_loss‚3.217864513397217 trainer/global_step‚2499 /loss/SRqqt4/assignment_loss‚1.4826791609327 /loss/FBt2/detection_loss‚0.6719537699237874 _step‚205eC]ä× 2loss/SRqqt2/assignment_loss‚1.0116924549846982 1loss/SRqqt3/assignment_loss‚1.114965204579326 1loss/FBt1/assignment_loss‚0.22137576739255566 1loss/FBt3/assignment_loss‚0.33068018885595457 0loss/SRqqt4/detection_loss‚2.761251655717691 %loss/total_loss‚43.82246839184083 % _timestamp‚1.7885133117592833e+09 /loss/FRt4/assignment_loss‚3.304212176281473 /loss/FRt1/detection_loss‚2.4049412007087336 0loss/SRqqt1/detection_loss‚2.155673865662065 1loss/SRqqt3/detection_loss‚1.7822335863915773 .loss/FBt1/detection_loss‚2.652783566713333 epoch‚2 0loss/FRt1/assignment_loss‚3.9158460358355907 0loss/FRt2/assignment_loss‚1.5558652457068949 1loss/SRqqt2/detection_loss‚0.8964392006015166 /loss/FBt4/detection_loss‚1.9700902104377747 1loss/FBt2/assignment_loss‚0.11356448443315718 1loss/FBt4/assignment_loss‚0.39544105477375524 .loss/FRt4/detection_loss‚3.217864513397217 trainer/global_step‚2499 /loss/SRqqt4/assignment_loss‚1.4826791609327 /loss/FBt2/detection_loss‚0.6719537699237874 _step‚205 0loss/FRt3/assignment_loss‚3.2413673043251037 /loss/FBt3/detection_loss‚2.9105795877320424 _runtime‚2937.229431338 1loss/SRqqt1/assignment_loss‚1.727653979605367 /loss/FRt2/detection_loss‚1.4097465963340272 .loss/FRt3/detection_loss‚2.573567580514484ÍR1ƒã“å‚(:ˆ ¢˜êÔ÷·€ gpu.0.gpu‚44.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚37.0 gpu.0.powerWatts‚118.142' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚29.5355 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0æ* ^æ‚(:Ó ¢˜êÔÌÙì€ disk.sdb1.in‚0 disk.sda1.in‚0 disk.md0p1.in‚ 0.0703125 disk.sda2.in‚ 0.0390625 disk.rbd0.in‚0) proc.memory.availableMB‚ 1300.63671875 disk.dm-0.in‚ 213.4921875 disk.sda.in‚ 0.0390625 disk.md0.out‚ 323.05859375 cpu‚25.22915489776586 proc.cpu.threads‚17 disk.md0p1.out‚ 323.05859375 disk.sda.out‚ 335.408203125 disk.rbd0.out‚ 0.845703125 disk.nvme1n1.in‚ 184.96484375* proc.memory.percent‚18.319988250732422 network.recv‚973553 disk.dm-0.out‚ 32873.109375 disk.sdb1.out‚0 disk.sdb2.out‚ 335.408203125# disk.nvme0n1.out‚17869.91796875 disk.sdb.in‚0.03125 disk.sda2.out‚ 335.408203125 network.sent‚ 5.417197e+06 disk.nvme0n1.in‚ 28.52734375 disk.sdb.out‚ 335.408203125# disk.nvme1n1.out‚15003.19140625 disk.md0.in‚ 0.0703125 disk.rbd2.out‚ 0.783203125$ memory_percent‚92.06154346466064! proc.memory.rssMB‚ 3001.546875) disk./.usagePercent‚75.84478010608713 disk.sda1.out‚0 disk.rbd2.in‚0% disk./.usageGB‚2712.5767097473145 disk.sdb2.in‚0.03125†U”ç‚(:‰ ±˜êÔ‚þ gpu.0.gpu‚55.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚39.0 gpu.0.powerWatts‚163.597' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚40.89925 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0M´LQdè‚(:Ù ±˜êÔ­Ä÷þ disk.md0.in‚ 0.0703125 disk.sdb2.in‚0.03125 disk.nvme0n1.in‚ 28.52734375 disk.nvme1n1.out‚ 15147.84375 network.sent‚ 5.52439e+06% disk./.usageGB‚2712.5756912231445 disk.sda1.in‚0 disk.dm-0.in‚ 213.4921875! proc.memory.rssMB‚ 3001.546875 disk.md0p1.in‚ 0.0703125 disk.rbd0.in‚0 disk.rbd0.out‚ 0.845703125$ memory_percent‚92.06416606903076) proc.memory.availableMB‚ 1300.20703125 network.recv‚978309 disk.md0p1.out‚ 324.65234375 disk.sdb1.out‚0 proc.cpu.threads‚17 disk.sda1.out‚0 disk.sda.out‚337.06591796875 disk.sda2.in‚ 0.0390625 disk.nvme1n1.in‚ 184.96484375 disk.sdb1.in‚0 disk.rbd2.out‚ 0.783203125* proc.memory.percent‚18.319988250732422) disk./.usagePercent‚75.84475162772505 disk.dm-0.out‚33168.01953125 disk.sdb.in‚0.03125 disk.rbd2.in‚0! disk.sdb2.out‚337.06591796875# disk.nvme0n1.out‚18020.17578125 disk.sdb.out‚337.06591796875 disk.sda.in‚ 0.0390625! disk.sda2.out‚337.06591796875 disk.md0.out‚ 324.65234375 cpu‚25.22811931103474`’Dï”é‚(:‰ À˜êÔ÷•õý gpu.0.gpu‚46.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚37.0 gpu.0.powerWatts‚133.017' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚33.25425 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0`“áŸaê‚(:Ö À˜êÔÀîõþ) disk./.usagePercent‚75.84475162772505 disk.sda1.out‚0 disk.md0p1.in‚ 0.0703125 disk.dm-0.in‚ 213.4921875 disk.md0p1.out‚ 326.4140625 disk.rbd0.in‚0 disk.sdb1.out‚0 proc.cpu.threads‚17 disk.rbd0.out‚ 0.845703125 disk.rbd2.in‚0 network.recv‚982824 disk.sdb.in‚0.03125 disk.sdb.out‚338.88525390625 disk.sda.in‚ 0.0390625 disk.sda2.in‚ 0.0390625! disk.nvme1n1.out‚ 15292.296875 disk.md0.out‚ 326.4140625 disk.rbd2.out‚ 0.783203125 disk.sdb2.in‚0.03125 disk.md0.in‚ 0.0703125! proc.memory.rssMB‚ 3001.546875* proc.memory.percent‚18.319988250732422% disk./.usageGB‚2712.5756912231445" disk.nvme0n1.out‚ 18020.1796875 disk.sda.out‚338.88525390625! disk.sda2.out‚338.88525390625 disk.nvme1n1.in‚ 184.96484375 disk.sdb1.in‚0$ memory_percent‚92.06266403198242 cpu‚25.227144176922295! disk.sdb2.out‚338.88525390625 disk.sda1.in‚0 disk.dm-0.out‚ 33312.4765625' proc.memory.availableMB‚ 1300.453125 disk.nvme0n1.in‚ 28.52734375 network.sent‚ 5.530351e+06ƒmrDë‚(:¹ ϘêÔä­ðþ disk.nvme0n1.in‚ 28.52734375 proc.cpu.threads‚17# disk.nvme1n1.out‚15292.52734375 disk.sda.in‚ 0.0390625 disk.sda2.in‚ 0.0390625 disk.rbd0.in‚0 disk.sdb2.out‚340.6875 disk.dm-0.in‚ 213.4921875 disk.sdb.in‚0.03125 cpu‚25.226179147776026 disk.rbd0.out‚ 0.845703125 disk.sda1.in‚0 disk.sda.out‚340.6875 disk.nvme1n1.in‚ 184.96484375! proc.memory.rssMB‚ 3001.546875) disk./.usagePercent‚75.83000378260911 disk.sda2.out‚340.6875 disk.md0.in‚ 0.0703125 disk.sda1.out‚0# disk.nvme0n1.out‚18120.30078125 disk.sdb1.out‚0 network.recv‚987094$ disk./.usageGB‚2712.048236846924 disk.md0.out‚ 328.1484375 disk.rbd2.out‚ 0.783203125 disk.dm-0.out‚ 33412.828125 disk.md0p1.in‚ 0.0703125# memory_percent‚92.0633316040039* proc.memory.percent‚18.319988250732422 disk.sdb1.in‚0 network.sent‚ 5.536281e+06 disk.rbd2.in‚0 disk.sdb2.in‚0.03125 disk.md0p1.out‚ 328.1484375 disk.sdb.out‚340.6875& proc.memory.availableMB‚ 1300.34375Ñ5”’ì‚(:‡ ϘêÔ’€«ÿ gpu.0.gpu‚54.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚169.61' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.4025 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ëì2í- _wandbruntime‚2989  _runtime‚2989òkø‡ÆîÀ & lr-AdamW/pg1‚0.0008260032126925463 & lr-AdamW/pg2‚0.0008260032126925463 trainer/global_step‚2549 $ _timestamp‚1.788513364252589e+09 _runtime‚2989.723216433 _step‚206¹žªâËïÅ $ _timestamp‚1.788513364252589e+09 _runtime‚2989.723216433 _step‚206 & lr-AdamW/pg1‚0.0008260032126925463 & lr-AdamW/pg2‚0.0008260032126925463 trainer/global_step‚2549Îøãå2ð- _wandbruntime‚2989  _runtime‚2989ž™ùP1ñëO YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚209832638246400 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26087 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚48113418 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 trainer/global_step‚2549 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 116652445 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚622 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚237797550043136 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚208031553948343 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytòHÂ:À$es.small_pool.current‚ 241443328 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚205141848469104 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2899618294272 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚209832638246400 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚32754293 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177046528 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31685 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26089 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚212732256540672 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450776 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2889705479239 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚15359745 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2888808067823 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31930 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2900517701632 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1205 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3026580488192 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3026339044864 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2900517701632 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚32754295 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411416 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚212732256540672 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214661 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410304 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 % _timestamp‚1.7885133642538054e+09 _step‚207 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955590656 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚64003202 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _runtime‚ 2989.72458381 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚208030639497567 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚63977115 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 116678534 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 265383424 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0ó0º®ö'òðO L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚63977115 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 116678534 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 265383424 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚209832638246400 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26087 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚48113418 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 trainer/global_step‚2549 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 116652445 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚622 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚237797550043136 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚208031553948343 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 241443328 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚205141848469104 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2899618294272 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚209832638246400 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚32754293 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177046528 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31685 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26089 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚212732256540672 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450776 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2889705479239 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚15359745 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2888808067823 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31930 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2900517701632 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1205 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3026580488192 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3026339044864 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2900517701632 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚32754295 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411416 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚212732256540672 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214661 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410304 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 % _timestamp‚1.7885133642538054e+09 _step‚207 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955590656 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚64003202 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _runtime‚ 2989.72458381 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚208030639497567Ïs'¨2ó- _wandbruntime‚2990  _runtime‚2990z;ÓGü&ôöM JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 116696934 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 116723049 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 116696934 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚52695230 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899420672 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚52695230 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450984 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚205219828563232 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚52695228 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31685 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955590656 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚583 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚237888020984320 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚205219811523872 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚64001706 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚209912286544896 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚585 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 246495744 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 398533632 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚212813917063168 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚209912286544896 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1180 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 _runtime‚2990.706363936 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411624 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚212813917063168 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177046528 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917423616 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2901630518272 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚237887774488576 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 222555648 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚48132028 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3027509652992 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚212812999639552 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177046528 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26113 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚208110642703997 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚64027819 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚234860288775680 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007237 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚48131443 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1205 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26115 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚209912268541952 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚32766671 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚15365357 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214661 trainer/global_step‚2549 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚234860264835584 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2890814140765 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 116723049 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917423616 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2901630518272 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚237887774488576 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 222555648 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚48132028 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3027509652992 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚212812999639552 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177046528 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26113 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚208110642703997 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚64027819 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚234860288775680 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007237 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚48131443 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1205 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26115 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚209912268541952 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3276é@F¥6671 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚15365357 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214661 trainer/global_step‚2549 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚234860264835584 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2890814140765 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 116723049 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 116723049 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 116696934 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚52695230 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899420672 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚52695230 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450984 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚205219828563232 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚52695228 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31685 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955590656 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚583 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚237888020984320 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚205219811523872 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚64001706 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚209912286544896 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚585 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 246495744 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 398533632 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚212813917063168 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚209912286544896 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1180 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 _runtime‚2990.706363936ПêŒw2ö- _wandbruntime‚2990  _runtime‚2990ïÝ£ÈW÷Ñ 1loss/FBt2/assignment_loss‚0.23077170767282057 /loss/FRt3/assignment_loss‚3.102857900162538 0loss/FRt4/assignment_loss‚3.3656530930445743 1loss/FBt4/assignment_loss‚0.15745171302819383 /loss/FRt2/detection_loss‚1.5091155926678694 epoch‚2 trainer/global_step‚2549 2loss/SRqqt3/assignment_loss‚1.7019033228809184 2loss/SRqqt4/assignment_loss‚1.7191309054051676 1loss/FBt1/assignment_loss‚0.11852395536843688 -loss/FRt4/detection_loss‚3.13184601526994 .loss/FRt1/assignment_loss‚2.66212182153355 .loss/FRt3/detection_loss‚2.329810460408529 2loss/SRqqt1/assignment_loss‚1.6340134336799383 /loss/FRt1/detection_loss‚2.0812587949362666 0loss/SRqqt3/detection_loss‚1.906573918394067 .loss/FBt1/detection_loss‚2.608239495754242 .loss/FBt4/detection_loss‚2.725601576268673 _runtime‚2990.707920413 /loss/FRt2/assignment_loss‚1.480964437419293 2loss/SRqqt2/assignment_loss‚1.1577247981226346 1loss/SRqqt1/detection_loss‚1.8882363906502724 .loss/FBt3/detection_loss‚2.310281586647034 /loss/SRqqt2/detection_loss‚1.08758994375096 1loss/SRqqt4/detection_loss‚2.6809335187558205 % _timestamp‚1.7885133652377944e+09 _step‚209 1loss/FBt3/assignment_loss‚0.12502269789401624 /loss/FBt2/detection_loss‚0.5829878816109025 &loss/total_loss‚42.298614961326656XoG\øÖ 1loss/FBt1/assignment_loss‚0.11852395536843688 -loss/FRt4/detection_loss‚3.13184601526994 .loss/FRt1/assignment_loss‚2.66212182153355 .loss/FRt3/detection_loss‚2.329810460408529 2loss/SRqqt1/assignment_loss‚1.6340134336799383 /loss/FRt1/detection_loss‚2.0812587949362666 0loss/SRqqt3/detection_loss‚1.906573918394067 .loss/FBt1/detection_loss‚2.608239495754242 .loss/FBt4/detection_loss‚2.725601576268673 _runtime‚2990.707920413 /loss/FRt2/assignment_loss‚1.480964437419293 2loss/SRqqt2/assignment_loss‚1.1577247981226346 1loss/SRqqt1/detection_loss‚1.8882363906502724 .loss/FBt3/detection_loss‚2.310281586647034 /loss/SRqqt2/detection_loss‚1.08758994375096 1loss/SRqqt4/detection_loss‚2.6809335187558205 % _timestamp‚1.7885133652377944e+09 _step‚209 1loss/FBt3/assignment_loss‚0.12502269789401624 /loss/FBt2/detection_loss‚0.5829878816109025 &loss/total_loss‚42.298614961326656 1loss/FBt2/assignment_loss‚0.23077170767282057 /loss/FRt3/assignment_loss‚3.102857900162538 0loss/FRt4/assignment_loss‚3.3656530930445743 1loss/FBt4/assignment_loss‚0.15745171302819383 /loss/FRt2/detection_loss‚1.5091155926678694 epoch‚2 trainer/global_step‚2549 2loss/SRqqt3/assignment_loss‚1.7019033228809184 2loss/SRqqt4/assignment_loss‚1.7191309054051676Ñ …x¯’ù‚(:‡ Þ˜êԜ޲€ gpu.0.gpu‚32.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚36.0 gpu.0.powerWatts‚114.408' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.602 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ñ_ú‚(:Ô Þ˜êÔï‡ò‚ disk.sda.out‚342.3955078125 disk.rbd0.in‚0 disk.nvme1n1.in‚ 184.96484375 disk.sdb1.out‚0 disk.sdb2.in‚0.03125# disk.nvme0n1.out‚18120.37890625 disk.sdb.in‚0.03125 disk.sdb.out‚342.3955078125 network.recv‚991730 disk.sda2.out‚342.3955078125* proc.memory.percent‚18.319988250732422 disk.md0.out‚ 329.7890625 disk.md0p1.out‚ 329.7890625 disk.nvme1n1.out‚ 15341.28125 disk.sdb1.in‚0 disk.sda1.out‚0 disk.nvme0n1.in‚ 28.52734375 disk.md0.in‚ 0.0703125! proc.memory.rssMB‚ 3001.546875% disk./.usageGB‚2712.0479431152344 disk.rbd2.out‚ 0.783203125 disk.sdb2.out‚342.3955078125 disk.dm-0.out‚33461.66015625) proc.memory.availableMB‚ 1300.26953125 cpu‚25.225235536289794) disk./.usagePercent‚75.82999556974815 disk.sda.in‚ 0.0390625 disk.rbd2.in‚0 disk.sda1.in‚0 disk.dm-0.in‚ 213.4921875# memory_percent‚92.0637845993042 network.sent‚ 5.555633e+06 disk.sda2.in‚ 0.0390625 disk.rbd0.out‚ 0.845703125 disk.md0p1.in‚ 0.0703125 proc.cpu.threads‚17³A0b“û‚(:ˆ í˜êÔÿ gpu.0.gpu‚48.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚160.922' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚40.2305 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0BØWü‚(:Ì í˜êԫ↠network.sent‚ 5.650991e+06 disk.sdb.in‚0.03125 disk.rbd2.out‚ 0.783203125 disk.sdb2.out‚ 344.326171875* proc.memory.percent‚18.319988250732422) disk./.usagePercent‚75.83070784878164! disk.nvme1n1.out‚ 15341.515625 disk.sdb1.in‚0# proc.memory.availableMB‚1300.75 disk.sda2.out‚ 344.326171875 disk.md0.in‚ 0.0703125 disk.rbd2.in‚0 disk.sda1.in‚0 disk.md0p1.out‚ 331.66015625 disk.sda1.out‚0 disk.nvme0n1.in‚ 28.52734375 network.recv‚995332 disk.sdb.out‚ 344.326171875 disk.sda2.in‚ 0.0390625 cpu‚25.22423878817991 proc.cpu.threads‚17$ disk./.usageGB‚2712.073417663574 disk.sda.out‚ 344.326171875 disk.rbd0.out‚ 0.845703125# disk.nvme0n1.out‚18498.14453125! proc.memory.rssMB‚ 3001.546875 disk.sda.in‚ 0.0390625 disk.nvme1n1.in‚ 184.96484375 disk.sdb1.out‚0 disk.md0.out‚ 331.66015625 disk.dm-0.out‚33839.66015625 disk.rbd0.in‚0 disk.sdb2.in‚0.03125 disk.dm-0.in‚ 213.4921875 disk.md0p1.in‚ 0.0703125$ memory_percent‚92.06085205078125zÓP4”ý‚(:‰ ü˜êÔ·§õÿ gpu.0.gpu‚54.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚170.559' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.63975 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0w×)Seþ‚(:Ú ü˜êÔøÒº€# disk.nvme0n1.out‚18650.54296875 disk.nvme1n1.in‚ 184.96484375 network.sent‚ 5.654834e+06 disk.sda.out‚345.93994140625) proc.memory.availableMB‚ 1300.58984375) disk./.usagePercent‚75.83077643150379 disk.sda1.out‚0 disk.md0p1.out‚ 333.2109375 disk.rbd2.in‚0 disk.sdb2.in‚0.03125 proc.cpu.threads‚17$ disk./.usageGB‚2712.075870513916 disk.dm-0.out‚33992.22265625 disk.sda2.in‚ 0.0390625 disk.rbd0.out‚ 0.845703125" disk.nvme1n1.out‚ 15341.6796875 disk.sdb1.in‚0 disk.md0.out‚ 333.2109375$ memory_percent‚92.06182956695557 disk.sda1.in‚0 disk.nvme0n1.in‚ 28.52734375 disk.rbd0.in‚0 disk.sdb1.out‚0 disk.md0.in‚ 0.0703125! proc.memory.rssMB‚ 3001.546875 network.recv‚999703 disk.dm-0.in‚ 213.4921875 disk.sdb.out‚345.93994140625 disk.sda.in‚ 0.0390625 disk.sdb.in‚0.03125 disk.rbd2.out‚ 0.783203125! disk.sdb2.out‚345.93994140625* proc.memory.percent‚18.319988250732422 cpu‚25.223311746688804 disk.md0p1.in‚ 0.0703125! disk.sda2.out‚345.93994140625¿2yiÿ‚(:Þ ‹™êÔïõÒÿ disk.md0p1.out‚ 334.80859375! disk.nvme0n1.out‚ 18821.546875 cpu‚25.222316243818394 network.recv‚ 1.004277e+06 disk.sda.in‚ 0.0390625 disk.sda2.in‚ 0.0390625 disk.nvme1n1.in‚ 184.96484375 disk.rbd2.in‚0 disk.dm-0.out‚ 34163.5703125$ memory_percent‚96.03245258331299 network.sent‚ 5.663211e+06 disk.rbd0.in‚0 disk.rbd0.out‚ 0.845703125 disk.sdb1.in‚0 disk.sdb2.in‚0.03125 disk.dm-0.in‚ 213.4921875* proc.memory.percent‚18.319988250732422 disk.sdb1.out‚0 disk.rbd2.out‚ 0.783203125 disk.sda.out‚347.60595703125! disk.sdb2.out‚347.60595703125 disk.nvme0n1.in‚ 28.52734375 disk.sdb.out‚347.60595703125 proc.cpu.threads‚17! disk.sda2.out‚347.60595703125( proc.memory.availableMB‚ 650.04296875) disk./.usagePercent‚75.81872773781375$ disk./.usageGB‚2711.644950866699 disk.md0.out‚ 334.80859375 disk.sda1.out‚0 disk.sdb.in‚0.03125! proc.memory.rssMB‚ 3001.546875 disk.md0.in‚ 0.0703125 disk.sda1.in‚0" disk.nvme1n1.out‚ 15342.0234375 disk.md0p1.in‚ 0.0703125ùVL…”€‚(:‰ ‹™êÔþ°èÿ gpu.0.gpu‚41.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚159.419' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚39.85475 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0'’2- _wandbruntime‚3045  _runtime‚3045eäbÆ‚À trainer/global_step‚2599 % _timestamp‚1.7885134199333582e+09 _runtime‚ 3045.40381368 _step‚210 & lr-AdamW/pg1‚0.0008236260138864925 & lr-AdamW/pg2‚0.0008236260138864925ì9Á_˃Štrainer/global_step‚2599 % _timestamp‚1.7885134199333582e+09 _runtime‚ 3045.40381368 _step‚210 & lr-AdamW/pg1‚0.0008236260138864925 & lr-AdamW/pg2‚0.0008236260138864925Ò¾¨M2„- _wandbruntime‚3045  _runtime‚3045ÈVŽò'…ìO YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1180 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450776 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411416 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31685 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2944254724931 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31685 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚53670230 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410304 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚65210017 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177046528 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚49015848 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2945152136347 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2956172281856 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚614 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3084681603584 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚211986005311851 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214661 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 118906336 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31930 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚211985090861075 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚33373095 _runtime‚3045.405031765 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2955272874496 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007237 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚53670232 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚242321612195328 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚242321363589120 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3084456937472 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚216769425898496 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26087 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2956172281856 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚239236930591744 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚216770343308800 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚213814171026944 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177046528 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚65236104 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚216769425898496 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚33373093 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 trainer/global_step‚2599 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26089 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31930 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚65210017 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917410304 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚213814171026944 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1205 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26089 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚15642755 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 224666112 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1180 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450776 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411416 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31685 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2944254724931 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31685 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚53670230 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410304 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚65210017 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177046528 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚49015848 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2945152136347 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2956172281856 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚614 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3084681603584 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚211986005311851 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214661 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 118906336 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31930 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚211985090861075 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚33373095 _runtime‚3045.405031765 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2955272874496 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007237 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚53670232 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚242321612195328 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚242321363589120 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3084456937472 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚216769425898496 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26087 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2956172281856 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚239236930591744 ?9DeviceStatsMonitor.on_train_batch_start/num_alï5kÁloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚216770343308800 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚213814171026944 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177046528 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚65236104 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚216769425898496 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚33373093 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 trainer/global_step‚2599 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26089 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31930 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚65210017 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917410304 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚213814171026944 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1205 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26089 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚15642755 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 224666112 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 JDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31930 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26113 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚239326249812480 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3085629049856 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚53690128 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚65234608 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2957285098496 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚53690130 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 398533632 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177046528 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214661 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 118924736 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411624 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2946260797873 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚53690130 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚216851086420992 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2956385677824 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450984 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚212064179616521 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31685 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚612 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917423616 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007237 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917423616 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚216850168997376 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 118950851 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚65234608 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚213893783319552 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899420672 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 118924736 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1180 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚33385471 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955590656 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚209118833269632 _runtime‚3046.382365357 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚15648421 _step‚212 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2957285098496 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚242411854922240 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2956385677824 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚49033890 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 242301440 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚213893801322496 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚209118816230272 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31930 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26113 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚239326249812480 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3085629049856 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚53690128 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚65234608 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2957285098496 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚53690130 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 398533632 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177046528 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214661 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 118924736 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411624 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2946260797873 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚53690130 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚216851086420992 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2956385677824 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450984 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚212064179616521 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31685 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚612 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917423616 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007237 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917423616 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚216850168997376 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 118950851 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚65234608 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚213893783319552 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899420672 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 118924736 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1180 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚33385471 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955590656 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚209118833269632 _runtime‚3046.382365357 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚15648421 _step‚212 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2957285098496 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚242411854922240 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2956385677824 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚49033890 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 242301440 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚213893801322496 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚209118816230272 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 FDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31685 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 _step‚215 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31930 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚220807512666624 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚33991893 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚220806595256320 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3010927454720 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3010927454720 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26089 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007237 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚637 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚212939840842544 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚54665132 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚215940456675359 % _timestamp‚1.7885134736238918e+09scY÷'˜ñO [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚54665130 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚246845421835776 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚220807512666624 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚635 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚246845672539136 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26089 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31685 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 _step‚215 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31930 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚220807512666624 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚33991893 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚220806595256320 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3010927454720 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3010927454720 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26089 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007237 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚637 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚212939840842544 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚54665132 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚215940456675359 % _timestamp‚1.7885134736238918e+09 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955590656 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407360 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955590656 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚15927512×:}Ä2™- _wandbruntime‚3100  _runtime‚3100YRnü&šöM T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚220888255778816 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3012040258048 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31930 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1180 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚243792210849280 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚15933760 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899420672 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177046528 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚220887338355200 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚34004271 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 121152538 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚631 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 398533632 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 trainer/global_step‚2649 _step‚216 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917423616 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚217875298097152 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚216018630980029 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚54685030 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955590656 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚49937398 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3143938827264 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1205 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3000810043357 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚66467510 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917423616 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3012939678720 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚54685030 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚49938031 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450984 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚213017837976032 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 121178653 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955590656 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899420672 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177046528 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚54685028 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26113 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚213017820936672 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚66493623 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚217875298097152 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3012040258048 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 _runtime‚3100.063375516 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214661 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 121178653 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚217875316100096 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411624 % _timestamp‚1.7885134745923016e+09 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3012939678720 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 121152538 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31685 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚246935898986496 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007237 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26113 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171ž† !6 ›ûM TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 250690048 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 226749952 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3001707454981 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚243792186909184 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3143712077312 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955590656 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899420672 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177046528 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚54685028 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26113 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚213017820936672 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚66493623 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚217875298097152 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3012040258048 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 _runtime‚3100.063375516 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214661 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 121178653 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.l{•dªËarge_pool.peak‚35 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚217875316100096 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411624 % _timestamp‚1.7885134745923016e+09 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3012939678720 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 121152538 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31685 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚246935898986496 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007237 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26113 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚220888255778816 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3012040258048 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31930 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1180 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚243792210849280 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚15933760 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899420672 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177046528 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚220887338355200 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚34004271 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 121152538 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚631 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 398533632 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 trainer/global_step‚2649 _step‚216 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917423616 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚217875298097152 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚216018630980029 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚54685030 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955590656 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚49937398 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3143938827264 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1205 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3000810043357 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚66467510 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917423616 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3012939678720 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚54685030 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚49938031 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450984 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚213017837976032 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 121178653 F)Ë©Å _runtime‚ 3150.75294583 _step‚218 & lr-AdamW/pg1‚0.0008186253168230534 & lr-AdamW/pg2‚0.0008186253168230534 trainer/global_step‚2699 % _timestamp‚1.7885135252825677e+09Úè¸X2ª- _wandbruntime‚3150  _runtime‚3150"]I©ò'«ìO N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917410304 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407360 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚221777182579200 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3066582034944 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚67701908 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚219893993588091 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26087 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚67675821 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚224843764614144 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955590656 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚16212819 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214661 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚55660030 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚248168852665344 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 123361940 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚55660032 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 230957568 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚216838845548944 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3055148039147 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚221777200582144 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚34610695 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚251369674435584 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚219894908038867 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3201076667904 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3200845710336 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚224844682024448 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚251369929333248 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚224843764614144 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚55660032 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1180 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26089 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 254897664 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3067481442304 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚50824132 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 398533632 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26089 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177046528 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚216838862588304 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31685 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚224844682024448 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955590656 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177046528 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31685É‘¿ö¬ñO QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410304 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚50823512 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31930 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚620 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407360 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚67675821 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26087 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3066582034944 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚16213437 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3056045450563 _step‚219 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚67701908 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚224843764614144 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚55660032 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1180 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26089 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 254897664 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3067481442304 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚50824132 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 398533632 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26089 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177046528 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚216838862588304 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31685 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚224844682024448 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955590656 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177046528 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31685 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917410304 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407360 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚221777182579200 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3066582034944 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚67701908 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LrAZ-FDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚219893993588091 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26087 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚67675821 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚224843764614144 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955590656 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚16212819 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214661 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚55660030 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚248168852665344 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 123361940 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚55660032 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 230957568 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚216838845548944 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3055148039147 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚221777200582144 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚34610695 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚251369674435584 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚219894908038867 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3201076667904 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3200845710336 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚224844682024448 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚251369929333248 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 JDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917423616 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3057154112089 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26115 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚216916842682432 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚248258171886080 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26113 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 214167040 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1180 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚67726525 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚224925425136640 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177046528 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚50841547 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚34623069 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚219973996794521 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411624 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31930 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450984 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 123406455 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 123380340 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚55679930 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚251460176296448 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007237 _step‚220 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177046528 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚221856830877696 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214661 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚2699 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚55679930 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955590656 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3056256700465 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚55679928 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31685 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917423616 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3057154112089 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26115 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚216916842682432 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚248258171886080 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26113 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 214167040 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1180 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚67726525 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚224925425136640 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177046528 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚50841547 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚34623069 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚219973996794521 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411624 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31930 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450984 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 123406455 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 123380340 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚55679930 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚251460176296448 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007237 _step‚220 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177046528 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚221856830877696 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214661 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚2699 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚55679930 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955590656 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3056256700465 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171Ü<ùA¶2°- _wandbruntime‚3151  _runtime‚3151šà¿+U±Ï 0loss/SRqqt3/detection_loss‚1.894175270497799 %loss/total_loss‚43.39031009414944 /loss/FRt1/assignment_loss‚3.056137741804123 0loss/FBt3/assignment_loss‚0.3722907692317676 _runtime‚3151.713004083 2loss/SRqqt3/assignment_loss‚1.4432576493918896 .loss/FRt4/detection_loss‚2.885284929564505 0loss/SRqqt2/detection_loss‚1.150338982231915 .loss/FRt3/assignment_loss‚3.44172643603019 .loss/FBt4/detection_loss‚2.762129076889583 _step‚221 1loss/FBt4/assignment_loss‚0.26367948623374104 /loss/FRt2/detection_loss‚1.6551931024952369 1loss/SRqqt1/detection_loss‚1.9414285136593712 epoch‚2 2loss/SRqqt4/assignment_loss‚1.3695467680692672 1loss/FBt1/assignment_loss‚0.11570779013305178 .loss/FBt2/detection_loss‚0.601812364196921 /loss/FBt3/detection_loss‚3.0635801255702972 $ _timestamp‚1.788513526242888e+09 .loss/FBt1/detection_loss‚2.371637225151062 trainer/global_step‚2699 0loss/FRt2/assignment_loss‚1.1074985569173639 /loss/FRt4/assignment_loss‚4.050816944151213 2loss/SRqqt1/assignment_loss‚1.3388555743628079 1loss/FBt2/assignment_loss‚0.23340560956929435 /loss/FRt3/detection_loss‚2.2788456588421226 2loss/SRqqt2/assignment_loss‚0.9245439896921572 0loss/SRqqt4/detection_loss‚3.050047966837883 .loss/FRt1/detection_loss‚2.018369562625885#îZ²Ô 2loss/SRqqt3/assignment_loss‚1.4432576493918896 .loss/FRt4/detection_loss‚2.885284929564505 0loss/SRqqt2/detection_loss‚1.150338982231915 .loss/FRt3/assignment_loss‚3.44172643603019 .loss/FBt4/detection_loss‚2.762129076889583 _step‚221 1loss/FBt4/assignment_loss‚0.26367948623374104 /loss/FRt2/detection_loss‚1.6551931024952369 1loss/SRqqt1/detection_loss‚1.9414285136593712 epoch‚2 2loss/SRqqt4/assignment_loss‚1.3695467680692672 1loss/FBt1/assignment_loss‚0.11570779013305178 .loss/FBt2/detection_loss‚0.601812364196921 /loss/FBt3/detection_loss‚3.0635801255702972 $ _timestamp‚1.788513526242888e+09 .loss/FBt1/detection_loss‚2.371637225151062 trainer/global_step‚2699 0loss/FRt2/assignment_loss‚1.1074985569173639 /loss/FRt4/assignment_loss‚4.050816944151213 2loss/SRqqt1/assignment_loss‚1.3388555743628079 1loss/FBt2/assignment_loss‚0.23340560956929435 /loss/FRt3/detection_loss‚2.2788456588421226 2loss/SRqqt2/assignment_loss‚0.9245439896921572 0loss/SRqqt4/detection_loss‚3.050047966837883 .loss/FRt1/detection_loss‚2.018369562625885 0loss/SRqqt3/detection_loss‚1.894175270497799 %loss/total_loss‚43.39031009414944 /loss/FRt1/assignment_loss‚3.056137741804123 0loss/FBt3/assignment_loss‚0.3722907692317676 _runtime‚3151.713004083ÝSª‚’³‚(:‡ ƒšêÔÓÔòþ gpu.0.gpu‚42.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚36.0 gpu.0.powerWatts‚114.408' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.602 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Èh2e´‚(:Ú ƒšêÔáÊ’ÿ network.recv‚ 1.039734e+06 disk.md0p1.out‚ 348.11328125 disk.sda.in‚ 0.0390625 disk.sda2.out‚361.3896484375 disk.sda1.in‚0$ disk./.usageGB‚2712.077449798584 disk.dm-0.in‚ 213.4921875! proc.memory.rssMB‚ 3001.546875 disk.dm-0.out‚35180.70703125 cpu‚25.21482431874081 disk.nvme0n1.in‚ 28.52734375 disk.sda.out‚361.3896484375 disk.rbd0.out‚ 0.845703125 disk.nvme1n1.in‚ 184.96484375 disk.sdb1.out‚0 disk.rbd2.in‚0 proc.cpu.threads‚17 disk.rbd2.out‚ 0.783203125" disk.nvme0n1.out‚ 19440.7265625 disk.sda2.in‚ 0.0390625# disk.nvme1n1.out‚15739.98046875 disk.sda1.out‚0* proc.memory.percent‚18.319988250732422( disk./.usagePercent‚75.8308205889641 disk.md0p1.in‚ 0.0703125 disk.sdb1.in‚0$ memory_percent‚96.03476524353027 disk.sdb.in‚0.03125 disk.rbd0.in‚0 disk.md0.out‚ 348.11328125 disk.sdb2.in‚0.03125 disk.sdb2.out‚361.3896484375 disk.sdb.out‚361.3896484375 disk.md0.in‚ 0.0703125' proc.memory.availableMB‚ 649.6640625 network.sent‚ 5.911453e+06R©P”µ‚(:‰ ’šêÔÒ˸ÿ gpu.0.gpu‚33.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚168.383' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.09575 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0—N°Ïa¶‚(:Ö ’šêÔŸ‘€% proc.memory.availableMB‚ 649.78125 network.recv‚ 1.045485e+06 disk.sda.out‚ 363.099609375 disk.sdb2.in‚0.03125 disk.sda1.in‚0) disk./.usagePercent‚75.83253558366009" disk.nvme0n1.out‚ 19500.8828125 disk.sdb.in‚0.03125 disk.md0.in‚ 0.0703125 disk.dm-0.in‚ 213.4921875 disk.dm-0.out‚35341.05078125 cpu‚25.214036876664753$ disk./.usageGB‚2712.138786315918 disk.rbd0.in‚0 disk.sda2.in‚ 0.0390625 disk.nvme1n1.in‚ 184.96484375 disk.md0.out‚ 349.74609375 disk.rbd2.out‚ 0.783203125 disk.sdb2.out‚ 363.099609375* proc.memory.percent‚18.319988250732422 disk.sdb.out‚ 363.099609375 disk.sda2.out‚ 363.099609375 disk.sda1.out‚0! proc.memory.rssMB‚ 3001.546875 proc.cpu.threads‚17 disk.md0p1.out‚ 349.74609375 disk.sdb1.in‚0$ memory_percent‚96.03404998779297 disk.rbd2.in‚0 network.sent‚ 6.019521e+06 disk.md0p1.in‚ 0.0703125 disk.nvme0n1.in‚ 28.52734375 disk.sda.in‚ 0.0390625 disk.rbd0.out‚ 0.845703125# disk.nvme1n1.out‚15840.16796875 disk.sdb1.out‚0ëÆëV“·‚(:ˆ ¡šêÔþšäÿ gpu.0.gpu‚54.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚158.746' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚39.6865 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0~8<ªi¸‚(:Þ ¡šêÔÅæ£ network.sent‚ 6.024768e+06) disk./.usagePercent‚75.83777816213492 disk.nvme0n1.in‚ 28.52734375 disk.sda2.in‚ 0.0390625 disk.rbd0.out‚ 0.845703125 disk.sda1.out‚0' proc.memory.availableMB‚ 650.1328125 cpu‚25.21317643899249 disk.sdb.in‚0.03125 disk.md0.out‚ 351.13671875 disk.sda1.in‚0! proc.memory.rssMB‚ 3001.546875! disk.sda2.out‚364.56591796875 disk.rbd0.in‚0 disk.sdb1.in‚0* proc.memory.percent‚18.319988250732422$ disk./.usageGB‚2712.326286315918# disk.nvme0n1.out‚19587.91796875 disk.rbd2.out‚ 0.783203125 disk.sdb2.in‚0.03125 disk.dm-0.in‚ 213.4921875 disk.md0p1.in‚ 0.0703125 disk.nvme1n1.in‚ 184.96484375 disk.md0.in‚ 0.0703125! disk.sdb2.out‚364.56591796875 proc.cpu.threads‚17 network.recv‚ 1.048765e+06 disk.sda.out‚364.56591796875 disk.md0p1.out‚ 351.13671875 disk.sdb.out‚364.56591796875# disk.nvme1n1.out‚15990.78515625 disk.sdb1.out‚0$ memory_percent‚96.03190422058105 disk.sda.in‚ 0.0390625 disk.rbd2.in‚0 disk.dm-0.out‚ 35578.703125ô2¹- _wandbruntime‚3205  _runtime‚3205ç[ۺǺÁ _runtime‚3205.382007458 _step‚222 & lr-AdamW/pg1‚0.0008160028607034408 & lr-AdamW/pg2‚0.0008160028607034408 trainer/global_step‚2749 % _timestamp‚1.7885135799115498e+09·ŽˆÌ»Æ % _timestamp‚1.7885135799115498e+09 _runtime‚3205.382007458 _step‚222 & lr-AdamW/pg1‚0.0008160028607034408 & lr-AdamW/pg2‚0.0008160028607034408 trainer/global_step‚2749Þ”m¿Å2¼- _wandbruntime‚3205  _runtime‚3205ýR³’½ìO [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3123136022528 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407360 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450776 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 =6DeviceStatsMonitor.on_train_batch_start/max_split_øu¨`&size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955590656 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31685 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚616 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚228880933971968 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31685 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚225758715359744 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚68908723 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 _step‚223 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1205 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 _runtime‚3205.383299124 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚223848444951599 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177046528 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚56654930 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3122236615168 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚51728173 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚35229495 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚618 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955590656 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31930 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚56654932 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚51727555 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚223849359402375 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚225758697356800 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚35229493 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚252634813702144 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚228880933971968 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚225758715359744 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚16498678 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411416 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 125589742 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1180 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚56654932 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410304 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3123136022528 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚220737867294704 % _timestamp‚1.7885135799127545e+09 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26089 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917410304 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚255893691232768 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚220737850255344 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚2749 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31930 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚228881851382272 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26089 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171ßQ½÷'¾ñO JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚618 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955590656 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31930 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚56654932 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚51727555 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚223849359402375 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚225758697356800 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚35229493 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚252634813702144 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚228880933971968 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚225758715359744 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚16498678 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411416 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 125589742 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1180 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚56654932 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410304 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3123136022528 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚220737867294704 % _timestamp‚1.7885135799127545e+09 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26089 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917410304 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚255893691232768 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚220737850255344 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚2749 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31930 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚228881851382272 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26089 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3123136022528 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407360 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450776 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955590656 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31685 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚616 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚228880933971968 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31685 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚225758715359744 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚68908723 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 _step‚223 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1205 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 _runtime‚3205.383299124 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚223848444951599 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177046528 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚56654930 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3122236615168 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚51728173 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚35229495߆¹2¿- _wandbruntime‚3206  _runtime‚3206ÎL¢ü&ÀöM WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚228962594494464 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚610 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚252724132922880 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚56674830 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1180 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 125634257 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177046528 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚225838327652352 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 125634257 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899420672 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚51746097 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚608 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚223928448158029 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3111703357573 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚225838345655296 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26113 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31685 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214661 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚225838345655296 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 trainer/global_step‚2749 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31930 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚56674828 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚68959427 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917423616 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 398533632 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3260055894528 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177046528 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚220815847388832 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917423616 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚255984419761664 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 125608142 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3260286838784 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3112600769197 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚56674830 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚56674828 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3123349418496 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚51745487 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31930 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007237 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚68933314 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955590656 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚68933314 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3124248839168 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚255984164877312 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450984 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _runtime‚3206.437852689 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955590656 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 % _timestamp‚1.7885135809668505e+09 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚228962594494464 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚228961677070848 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 230944256 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚225838327652352 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899420672 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3123349418496 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚252724108982784 _step‚224 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 125608142 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚223927533707045d²ÜX ÁûM L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917423616 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 398533632 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3260055894528 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177046528 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚220815847388832 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917423616 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚255984419761664 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 125608142 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3260286838784 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3112600769197 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚56674830 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚56674828 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3123349418496 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚51745487 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31930 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007237 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚68933314 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955590656 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚68933314 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3124248839168 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚255984164877312 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450984 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.aÜ T©ll.current‚ 14220787712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _runtime‚3206.437852689 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955590656 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 % _timestamp‚1.7885135809668505e+09 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚228962594494464 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚228961677070848 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 230944256 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚225838327652352 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899420672 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3123349418496 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚252724108982784 _step‚224 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 125608142 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚223927533707045 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚228962594494464 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚610 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚252724132922880 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚56674830 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1180 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 125634257 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177046528 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚225838327652352 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 125634257 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899420672 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚51746097 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚608 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚223928448158029 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3111703357573 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚225838345655296 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26113 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31685 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214661 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚225838345655296 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 trainer/global_step‚2749 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31930 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚56674828 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚68959427 EDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31685 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1205 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26089 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450776 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚57649832 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚57649830 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚70141625 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26087 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚232918103329792 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 248606208 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917410304 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411416 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚70167712 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚229740230137344 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚70167712 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚232918103329792 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚229740230137344 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3178790602752 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚618 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚232919020740096 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚616 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214661 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007237 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚70141625 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 224666112 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26087 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚232919020740096 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407360 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚260417876029440 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚224636854961744 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955590656 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3177891195392 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177046528 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚229740212134400 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31685 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1205 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26089 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450776 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚57649832 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚57649830 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚70141625 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26087 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚232918103329792 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 248606208 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917410304 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411416 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚70167712 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚229740230137344 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚70167712 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚232918103329792 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚229740230137344 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3178790602752 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚618 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚232919020740096 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚616 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214661 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007237 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚70141625 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 224666112 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26087 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚232919020740096 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407360 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚260417876029440 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚224636854961744 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_starÂlÅØt/active_bytes.all.peak‚ 13955590656 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3177891195392 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177046528 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚229740212134400 JDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214661 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177046528 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955590656 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚52649011 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚16788342 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 220458496 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3168047426305 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411624 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 _step‚228 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚224714852095232 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3179903419392 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26113 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚260508121488384 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450984 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899420672 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917423616 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚224714835055872 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26115 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 127862059 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚227881985070553 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917423616 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31930 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚70166216 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚70166216 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3318051468800 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 244398592 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 trainer/global_step‚2799 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955590656 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚232999763852288 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26113 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚229819860432896 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31685 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚52649635 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚229819860432896 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚232999763852288 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1180 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚260508365886976 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚57669728 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007237F›K³ÿ&ÕùM H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚35860671 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1205 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚622 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3318271927296 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955590656 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚52649011 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚16788342 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 220458496 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3168047426305 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411624 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 _step‚228 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚224714852095232 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3179903419392 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26113 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚260508121488384 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450984 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899420672 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917423616 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚224714835055872 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26115 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 127862059 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚227881985070553 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917423616 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31930 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚70166216 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚70166216 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3318051468800 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 244398592 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 trainer/global_step‚2799 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955590656 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚232999763852288 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26113 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚229819860432896 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31685 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚52649635 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚229819860432896 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚232999763852288 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1180 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚260508365886976 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚57669728 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007237 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177046528 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3179003998720 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚16788964 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214661 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177046528 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚236955272687616 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1180 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 398533632 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 130019257 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚264941938599424 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410304 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚53536695 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1205 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 trainer/global_step‚2849 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3221488010471 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450776 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚58644730 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚71400614 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955590656 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407360 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31930 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 250703360 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 226763264 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31685 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214661 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3233545775616 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚634 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _step‚231 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26089 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚58644730 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31930 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407360 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 130045346 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007237 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚58644732 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917410304 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3234445182976 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 _runtime‚3313.467329399 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚636 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚228535859668144 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26089 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚264941687896064 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚71374527 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31685 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214661 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3233545775616 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚634 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _step‚231 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26089 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚58644730 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31930 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407360 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 130045346 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007237 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚58644732 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917410304 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3234445182976 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 _runtime‚3313.467329399 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚636 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚228535859668144 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26089 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚264941687896064 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚71374527 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚236955272687616 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1180 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 398533632 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0ç+?˜þ2å- _wandbruntime‚3314  _runtime‚3314’«ü&æöM L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917423616 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26115 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚261656031056384 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚3314.678519707 _step‚232 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚265032171020800 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚228613856801632 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 130063746 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚233801357207552 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 398533632 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177046528 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚17075941 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3376360422912 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899420672 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 220458496 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26113 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26113 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚630 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚53554782 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955590656 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚58664628 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3234658578944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31930 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚71399118 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚71425231 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31685 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 244398592 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚58664630 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚71399118 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚233801375210496 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 130089861 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 130089861 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚36479471 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3222596671789 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177046528 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450984 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31930 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1205 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007237 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3235557999616 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214661 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚233801375210496 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 130063746 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚53555412 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚231837350885045 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚58664628 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚233801357207552 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955590656 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚17075313 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚265032415419392 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚261656054996480 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3223494083413 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚237036933210112 % _timestamp‚1.7885136892072656e+09v=Mb™ çûM XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3222596671789 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177046528 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450984 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31930 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1205 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007237 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3235557999616 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214661 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚233801375210496 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 130063746 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚5ä7ݸh3555412 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚231837350885045 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚58664628 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚233801357207552 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955590656 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚17075313 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚265032415419392 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚261656054996480 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3223494083413 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚237036933210112 % _timestamp‚1.7885136892072656e+09 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917423616 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26115 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚261656031056384 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚3314.678519707 _step‚232 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚265032171020800 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚228613856801632 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 130063746 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚233801357207552 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 398533632 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177046528 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚17075941 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3376360422912 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899420672 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 220458496 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26113 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26113 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚630 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚53554782 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955590656 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚58664628 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3234658578944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31930 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚71399118 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚71425231 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31685 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 244398592 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚58664630 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚71399118 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚233801375210496 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 130089861 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 130089861 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚36479471è©ËÔ€2è- _wandbruntime‚3314  _runtime‚3314FkíCZéÔ 2loss/SRqqt1/assignment_loss‚2.0426155238174926 /loss/FRt1/detection_loss‚2.2142648758684778 .loss/FBt4/detection_loss‚3.383173704147339 0loss/FRt2/assignment_loss‚1.5325851738452911 /loss/FBt2/detection_loss‚0.5409417715668678 2loss/SRqqt2/assignment_loss‚0.8073937351666931 0loss/SRqqt2/detection_loss‚0.899818919815015 1loss/SRqqt3/detection_loss‚1.8628741665831152 trainer/global_step‚2849 /loss/FRt4/detection_loss‚3.0528834526027953 1loss/SRqqt4/detection_loss‚2.4970109837395804 _runtime‚3314.680018384 _step‚233 epoch‚2 1loss/SRqqt3/assignment_loss‚1.151805697254977 1loss/FBt3/assignment_loss‚0.26905859850157865 2loss/FBt4/assignment_loss‚0.058112460043048486 1loss/SRqqt1/detection_loss‚1.8194263752769022 $loss/total_loss‚44.4439650394375 0loss/FBt1/assignment_loss‚0.6900835131794641 1loss/FBt2/assignment_loss‚0.16428045777703212 /loss/FRt3/detection_loss‚2.3230685196492984 0loss/FRt3/assignment_loss‚2.7628329163012295 /loss/FRt2/detection_loss‚1.6848902738868043 .loss/FBt1/detection_loss‚3.427304724852244 .loss/FBt3/detection_loss‚2.320276769995689 0loss/FRt1/assignment_loss‚2.8900194546071494 -loss/FRt4/assignment_loss‚3.9273693050657 2loss/SRqqt4/assignment_loss‚2.1218736658937165 % _timestamp‚1.7885136892098224e+09NG-´_êÙ /loss/FRt1/detection_loss‚2.2142648758684778 .loss/FBt4/detection_loss‚3.383173704147339 0loss/FRt2/assignment_loss‚1.5325851738452911 /loss/FBt2/detection_loss‚0.5409417715668678 2loss/SRqqt2/assignment_loss‚0.8073937351666931 0loss/SRqqt2/detection_loss‚0.899818919815015 1loss/SRqqt3/detection_loss‚1.8628741665831152 trainer/global_step‚2849 /loss/FRt4/detection_loss‚3.0528834526027953 1loss/SRqqt4/detection_loss‚2.4970109837395804 _runtime‚3314.680018384 _step‚233 epoch‚2 1loss/SRqqt3/assignment_loss‚1.151805697254977 1loss/FBt3/assignment_loss‚0.26905859850157865 2loss/FBt4/assignment_loss‚0.058112460043048486 1loss/SRqqt1/detection_loss‚1.8194263752769022 $loss/total_loss‚44.4439650394375 0loss/FBt1/assignment_loss‚0.6900835131794641 1loss/FBt2/assignment_loss‚0.16428045777703212 /loss/FRt3/detection_loss‚2.3230685196492984 0loss/FRt3/assignment_loss‚2.7628329163012295 /loss/FRt2/detection_loss‚1.6848902738868043 .loss/FBt1/detection_loss‚3.427304724852244 .loss/FBt3/detection_loss‚2.320276769995689 0loss/FRt1/assignment_loss‚2.8900194546071494 -loss/FRt4/assignment_loss‚3.9273693050657 2loss/SRqqt4/assignment_loss‚2.1218736658937165 % _timestamp‚1.7885136892098224e+09 2loss/SRqqt1/assignment_loss‚2.0426155238174926é=ÍnØ’ë‚(:‡ ™›êÔ߉ˆÿ gpu.0.gpu‚2.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚37.0 gpu.0.powerWatts‚160.922' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚40.2305 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0õe]ƒbì‚(:× ™›êÔÿÇÿ" disk.nvme0n1.out‚ 19955.3671875 disk.sdb.out‚ 378.498046875 disk.rbd0.out‚ 0.845703125 disk.sdb1.out‚0 disk.md0.in‚ 0.0703125 disk.md0p1.in‚ 0.0703125 network.recv‚ 1.082913e+06 disk.rbd2.in‚0 disk.rbd2.out‚ 0.783203125 disk.sdb2.in‚0.03125( proc.memory.availableMB‚ 649.77734375 disk.sdb.in‚0.03125" disk.nvme1n1.out‚ 16706.7109375 disk.dm-0.out‚ 36662.078125* proc.memory.percent‚18.319988250732422% disk./.usageGB‚2711.6978073120117 disk.nvme0n1.in‚ 28.52734375 disk.md0p1.out‚ 364.57421875 disk.sda1.in‚0 disk.dm-0.in‚ 213.4921875 network.sent‚ 6.274252e+06) disk./.usagePercent‚75.82020562614682 disk.sda2.in‚ 0.0390625 disk.sdb1.in‚0$ memory_percent‚96.03407382965088 cpu‚25.206443529568105 disk.sda.out‚ 378.498046875 disk.sda2.out‚ 378.498046875 disk.rbd0.in‚0 proc.cpu.threads‚17 disk.sda.in‚ 0.0390625 disk.nvme1n1.in‚ 184.96484375 disk.md0.out‚ 364.57421875 disk.sdb2.out‚ 378.498046875 disk.sda1.out‚0! proc.memory.rssMB‚ 3001.546875tüÉ”í‚(:‰ ¨›êÔÐןÿ gpu.0.gpu‚33.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚157.977' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚39.49425 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¦Õ;½cî‚(:Ø ¨›êÔŠëÒÿ disk.nvme0n1.in‚ 28.52734375 disk.sdb2.out‚379.8369140625 disk.sda1.in‚0 disk.md0p1.out‚ 365.85546875 disk.sdb.out‚379.8369140625( proc.memory.availableMB‚ 649.86328125 disk.dm-0.in‚ 213.4921875 disk.md0p1.in‚ 0.0703125* proc.memory.percent‚18.319988250732422# disk.nvme0n1.out‚19955.43359375 network.sent‚ 6.382216e+06 network.recv‚ 1.088091e+06 disk.nvme1n1.in‚ 184.96484375 proc.cpu.threads‚17 disk.md0.out‚ 365.85546875) disk./.usagePercent‚75.82380957886667 disk.sdb2.in‚0.03125 disk.sda.in‚ 0.0390625# disk.nvme1n1.out‚16868.41015625 disk.sdb1.in‚0 disk.sdb1.out‚0# disk./.usageGB‚2711.82670211792 disk.dm-0.out‚ 36823.84375 disk.sda2.in‚ 0.0390625 disk.md0.in‚ 0.0703125 disk.rbd2.in‚0! proc.memory.rssMB‚ 3001.546875 disk.sda1.out‚0 disk.sda2.out‚379.8369140625 disk.rbd0.in‚0 disk.rbd2.out‚ 0.783203125 cpu‚25.2056681491617 disk.sdb.in‚0.03125 disk.sda.out‚379.8369140625 disk.rbd0.out‚ 0.845703125$ memory_percent‚96.03354930877686ÔÕsM”ï‚(:‰ ·›êÔáùý gpu.0.gpu‚39.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚37.0 gpu.0.powerWatts‚162.813' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚40.70325 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ñI²að‚(:Ö ·›êÔÕ±—ÿ disk.sda1.out‚0$ memory_percent‚96.03445529937744 disk.rbd2.in‚0 disk.sda.out‚ 381.462890625 disk.rbd0.out‚ 0.845703125 disk.nvme1n1.in‚ 184.96484375 disk.sdb1.out‚0 disk.md0.in‚ 0.0703125 disk.sda1.in‚0 disk.dm-0.out‚37012.49609375 disk.md0p1.in‚ 0.0703125# disk.nvme1n1.out‚17056.86328125 disk.md0.out‚ 367.421875 disk.sdb2.in‚0.03125 disk.sdb.out‚ 381.462890625 disk.rbd2.out‚ 0.783203125 disk.md0p1.out‚ 367.421875 network.sent‚ 6.388068e+06 disk.sda.in‚ 0.0390625 disk.sdb1.in‚0! proc.memory.rssMB‚ 3001.546875 cpu‚25.204834310002678) disk./.usagePercent‚75.83427489695738 disk.nvme0n1.in‚ 28.52734375 disk.sdb.in‚0.03125( proc.memory.availableMB‚ 649.71484375* proc.memory.percent‚18.319988250732422 proc.cpu.threads‚17 disk.sda2.out‚ 381.462890625 disk.dm-0.in‚ 213.4921875" disk.nvme0n1.out‚ 19955.6328125 network.recv‚ 1.092299e+06% disk./.usageGB‚2712.2009925842285 disk.sda2.in‚ 0.0390625 disk.rbd0.in‚0 disk.sdb2.out‚ 381.462890625˜Ò|R“ñ‚(:ˆ Æ›êÔ«æ–þ gpu.0.gpu‚56.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚36.0 gpu.0.powerWatts‚115.966' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.9915 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¢W£iò‚(:Þ Æ›êÔº‹¿ÿ disk.rbd0.out‚ 0.845703125 cpu‚25.204143542320683 disk.sda2.in‚ 0.0390625 disk.nvme0n1.in‚ 28.52734375# disk.nvme0n1.out‚19955.63671875 network.sent‚ 6.394382e+06 network.recv‚ 1.096687e+06 disk.dm-0.in‚ 213.4921875$ memory_percent‚96.03641033172607! disk.sda2.out‚382.86669921875 disk.md0p1.out‚ 368.76953125 disk.sdb2.in‚0.03125 disk.md0p1.in‚ 0.0703125 disk.nvme1n1.in‚ 184.96484375 disk.sdb1.in‚0 disk.sdb1.out‚0) disk./.usagePercent‚75.83778381514314 disk.rbd0.in‚0# disk.nvme1n1.out‚17142.17578125 disk.md0.in‚ 0.0703125! disk.sdb2.out‚382.86669921875 disk.sdb.in‚0.03125 proc.cpu.threads‚17 disk.sda.in‚ 0.0390625 disk.md0.out‚ 368.76953125 disk.sda1.in‚0( proc.memory.availableMB‚ 649.39453125! proc.memory.rssMB‚ 3001.546875 disk.rbd2.out‚ 0.783203125 disk.sda1.out‚0 disk.dm-0.out‚ 37097.8125 disk.sdb.out‚382.86669921875* proc.memory.percent‚18.319988250732422 disk.rbd2.in‚0$ disk./.usageGB‚2712.326488494873 disk.sda.out‚382.86669921875þ¯rS2ó- _wandbruntime‚3374  _runtime‚3374.KÇôÁ _step‚234 & lr-AdamW/pg1‚0.0008076536702809765 & lr-AdamW/pg2‚0.0008076536702809765 trainer/global_step‚2899 % _timestamp‚1.7885137490514774e+09 _runtime‚3374.522102733ú¢`xÌõÆ & lr-AdamW/pg1‚0.0008076536702809765 & lr-AdamW/pg2‚0.0008076536702809765 trainer/global_step‚2899 % _timestamp‚1.7885137490514774e+09 _runtime‚3374.522102733 _step‚234êb6ÙŒ2ö- _wandbruntime‚3374  _runtime‚3374л‘Fò'÷ìO GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26087 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411416 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚37085895 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 224666112 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407360 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚72633516 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26087 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚237703259692544 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚240992442045440 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3290099763200 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚72607429 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚37085893 _step‚235 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 132247059 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚266032696812544 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚615 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3433123205632 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚237703241689600 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3289200355840 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚613 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 248606208 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚237703241689600 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3290099763200 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31930 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚235711799042123 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚269466044684288 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚269465796078080 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚232434864374544 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31930 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1180 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26089 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31685 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚17356097 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚17355484 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 % _timestamp‚1.7885137490530667e+09 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚240993359455744 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407360 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _runtime‚3374.524579684 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚240992442045440 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955590656 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177046528 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚59639630 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚232434881413904 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917410304 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26089 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚54441992 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚266032672872448 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚235712713492899 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450776 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3276934667579 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚59639632 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚240993359455744 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3277832078995 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007237 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3277832078995 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007237 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31685 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚17356097 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚17355484 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 % _timestamp‚1.7885137490530667e+09 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚240993359455744 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407360 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _runtime‚3374.524579684 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚240992442045440 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955590656 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177046528 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚59639630 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚232434881413904 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917410304 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26089 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚54441992 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚266032672872448 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚235712713492899 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450776 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3276934667579 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚59639632 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚240993359455744 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171ëê‰W”ù‚(:‰ Õ›êÔݾÿ gpu.0.gpu‚51.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚37.0 gpu.0.powerWatts‚115.025' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.75625 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0™2X5fú‚(:Û Õ›êÔžæéÿ disk.sda1.in‚0 disk.dm-0.out‚37444.22265625 disk.sda2.in‚ 0.0390625! disk.sda2.out‚384.86474609375 disk.sdb1.out‚0$ disk./.usageGB‚2712.576473236084 disk.sda1.out‚0 disk.sda.in‚ 0.0390625 disk.nvme1n1.in‚ 184.96484375 disk.md0.in‚ 0.0703125$ memory_percent‚96.03638648986816 cpu‚25.203384691808658 proc.cpu.threads‚17) disk./.usagePercent‚75.84477349313413 disk.nvme0n1.in‚ 28.52734375 disk.sdb.in‚0.03125 disk.rbd0.in‚0" disk.nvme1n1.out‚ 17488.4296875 disk.sdb1.in‚0 disk.rbd2.in‚0 disk.sdb2.in‚0.03125 network.sent‚ 6.39963e+06 disk.md0p1.in‚ 0.0703125 disk.md0p1.out‚ 370.703125 disk.rbd0.out‚ 0.845703125 disk.rbd2.out‚ 0.783203125' proc.memory.availableMB‚ 649.3984375! proc.memory.rssMB‚ 3001.546875* proc.memory.percent‚18.319988250732422 network.recv‚ 1.100177e+06 disk.dm-0.in‚ 213.4921875 disk.md0.out‚ 370.703125! disk.sdb2.out‚384.86474609375# disk.nvme0n1.out‚19955.79296875 disk.sdb.out‚384.86474609375 disk.sda.out‚384.864746093750Éõà2û- _wandbruntime‚3375  _runtime‚3375´4Šü&üöM >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3291212579840 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚615 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚72632020 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚54459431 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3291212579840 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 244398592 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚235791802248553 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚237782889988096 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚17361162 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3278940740521 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 _step‚236 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 132317663 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚235790887797569 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007237 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚59659530 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚237782871985152 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1205 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚72658133 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177046528 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚232512861508032 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 _runtime‚3375.612686981 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917423616 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31685 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚266122016033280 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177046528 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3434291123200 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚232512844468672 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955590656 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1180 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚266121992093184 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 220458496 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 132291548 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917423616 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214661 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚37098269 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚2899 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31930 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚241073185144320 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚613 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 132291548 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚17361775 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 398533632 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3278043328897 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171Â\Á'ýûM XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚59659528 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚269556527614976 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3434511581696 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 % _timestamp‚1.7885137501411273e+09 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 132317663 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3290313159168 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚59659528 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26113 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26115 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚72632020 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚54459431 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3291212579840 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 244398592 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚235791802248553 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚237782889988096 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚17361162 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3278940740521 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 _step‚236 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 132317663 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚235790887797569 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007237 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚59659530 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚237782871985152 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1205 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚72658133 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177046528 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚232512861508032 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 _runtime‚3375.612686981 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917423616 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31685 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚266122016033280 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177046528 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3434291123200 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚232512844468672 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955590656 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1180 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚266121992093184 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 220458496 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 132291548 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917423616 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214661 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚37098269 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚2899 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31930 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚241073185144320 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚613 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 132291548 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚17361775 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 398533632 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3278043328897 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3291212579840 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚615 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚72632020 PDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 134474861 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚60634532 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955590656 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 398533632 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31685 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚245029611403264 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410304 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407360 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 134500950 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26089 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31930 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚241684756467200 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3344854936064 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3345754343424 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31685 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3491530473984 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚245030528813568 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 254897664 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955590656 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚236333886120304 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 % _timestamp‚1.7885138077696133e+09A)}÷'ŒñO GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚73840331 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚273990188323328 trainer/global_step‚2949 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚55346947 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26087 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚73866418 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚270498657849344 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 134474861 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚60634532 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955590656 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 398533632 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31685 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚245029611403264 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410304 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407360 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 134500950 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26089 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31930 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚241684756467200 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3344854936064 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3345754343424 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31685 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3491530473984 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚245030528813568 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 254897664 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955590656 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚236333886120304 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 % _timestamp‚1.7885138077696133e+09 JDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26113 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚239746253612061 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411624 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚73864922 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177046528 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3492473582592 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3346867160064 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1205 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955590656 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 134519350 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚73864922 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007237 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚73891035 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚37717071 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚609 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚245111271925760 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 134519350 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917423616 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 % _timestamp‚1.7885138088241906e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31930 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚60654430 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚73891035 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3346867160064 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚245110354502144 _runtime‚3434.295786992 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31685 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚55365015 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚37717069 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1180 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚241764404765696 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚274080426712576 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 134545465 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 398533632 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214661 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955590656 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899420672 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 242301440 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3345967739392 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564íB_ÁûM >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚239746253612061 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411624 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚73864922 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177046528 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3492473582592 D>DeviceStatsMonitor.on_train_batch_end/reserved_b1'ž@$ytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3346867160064 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1205 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955590656 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 134519350 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚73864922 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007237 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚73891035 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚37717071 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚609 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚245111271925760 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 134519350 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917423616 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 % _timestamp‚1.7885138088241906e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31930 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚60654430 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚73891035 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3346867160064 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚245110354502144 _runtime‚3434.295786992 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31685 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚55365015 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚37717069 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1180 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚241764404765696 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚274080426712576 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 134545465 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 398533632 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214661 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955590656 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899420672 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 242301440 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3345967739392 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚241764404765696 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26113 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚270587977070080 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26113 FDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3401408923648 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177046528 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚278513967093760 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955590656 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚56253842 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚17930946 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚240232873787344 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917410304 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚240232890826704 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚75099320 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚245666289247744 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚274964594946048 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3549372147712 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31685 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955590656 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚599 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214661 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 136728752 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚61629432 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26087 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚249066780761088 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚75073233 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007237 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 136702663 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚75099320 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚38323493 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 218374656 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177046528 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚61629430 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚245666289247744 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410304 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚249066780761088 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407360 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 242314752 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚75073233 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1205 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 _step‚243 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407360 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 398533632 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3388725393211 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚61629430 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31685 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚274964618886144 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 trainer/global_step‚2999 % _timestamp‚1.7885138615263693e+09 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450776 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26089 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚245666271244800 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26089 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚38323495 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1u…íO“ ñO PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚75099320 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚245666289247744 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚274964594946048 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3549372147712 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31685 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955590656 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚599 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214661 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 136728752 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚61629432 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26087 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚249066780761088 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚75073233 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007237 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 136702663 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚75099320 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚38323493 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 218374656 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177046528 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚61629430 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚245666289247744 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410304 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚249066780761088 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407360 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 242314752 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 LDDeviceStatsMonitor.on_train_baE‘Èïdtch_start/segment.large_pool.allocated‚607 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚75073233 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1205 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 _step‚243 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407360 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 398533632 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3388725393211 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚61629430 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31685 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚274964618886144 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 trainer/global_step‚2999 % _timestamp‚1.7885138615263693e+09 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450776 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26089 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚245666271244800 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26089 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚38323495 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3401408923648 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177046528 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚278513967093760 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955590656 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚56253842 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚17930946 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚240232873787344 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917410304 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚240232890826704ó[j¡2¡- _wandbruntime‚3487  _runtime‚3487r™Ìü&¢öM N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955590656 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177046528 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 136773267 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 214167040 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007237 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26113 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3550548198400 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚275053938106880 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177046528 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚38335869 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚278604462365184 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3401622319616 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214661 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 136773267 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚245745919543296 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚588 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚75123937 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 238107136 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚61649330 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚245745901540352 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31685 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚2999 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3550762365440 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917423616 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚61649330 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚243700704975569 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚240310870920832 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚38335871 _runtime‚3487.963389993 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚249148441283584 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚249148441283584 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917423616 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚245745919543296 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚245745901540352 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 398533632 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411624 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚278604700472320 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚56271857 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 136747152 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚75123937åFÀ'£ûM S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚249147523859968 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3401622319616 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚61649328 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1205 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917423616 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚61649330 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚243700704975569 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚240310870920832 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚38335871 _runtime‚3487.963389993 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚249148441283584 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚249148441283584 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917423616 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚245745919543296 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚245745901540352 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 398533632 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411624 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚278604700472320 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚56271857 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 136747152 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚75123937 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955590656 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177046528 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 136773267 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 214167040 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007237 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26113 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3550548198400 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚275053938106880 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177046528 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚38335869 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚278604462365184 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3401622319616 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214661 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 136773267 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚245745919543296 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚588 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚75123937 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 238107136 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚61649330 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚245745901540352 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31685 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚2999 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3550762365440 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26089 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955590656 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1205 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 138956554 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1180 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3607642716160 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955590656 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 398533632 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31685 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410304 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 138930465 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3457063503872 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407360 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3607869479424 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214661 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚253104867529216 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31930 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚76332222 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007237 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚76332222 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3456164096512 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚249647786022400 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚640 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 226763264 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚76306135 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26087 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚253103950118912 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚62624330 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚62624332 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚638 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚247576067583423 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚244131878493744 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31930 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚249647804025344 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3457063503872 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚57160450 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 _runtime‚3542.826627102 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚253104867529216 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚38942295 _step‚247 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3443274638903 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚76306135 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚38942293 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0Tj'eö'´ðO M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955590656 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 398533632 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31685 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410304 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 138930465 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3457063503872 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407360 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3607869479424 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214661 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚253104867529216 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31930 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚76332222 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007237 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚76332222 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3456164096512 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚249647786022400 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚640 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 226763264 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚76306135 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26087 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚253103950118912 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚62624330 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚62624332 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚638 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚247576067583423 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚244131878493744 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31930 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚249647804025344 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3457063503872 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚57160450 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 _runtime‚3542.826627102 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚253104867529216 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚38942295 _step‚247 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3443274638903 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚76306135 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚38942293 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917410304 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26089 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955590656 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1205 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 138956554 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1180 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3607642716160 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984÷ßò­2µ- _wandbruntime‚3543  _runtime‚3543cðhå"¶õM FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 trainer/global_step‚3049 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 230944256 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 _runtime‚3543.964817728 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚247654241888093 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3458176320512 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3458176320512 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚283128922859520 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450984 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚654 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚279519875203584 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899420672 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚62644228 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917423616 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚38954669 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26115 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚76356839 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955590656 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚62644230 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚62644228 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917423616 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 138974954 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚244209875627232 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 254884352 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3608792771584 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚247655156339077 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007237 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚57178495 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚18224478 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3444383300221 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚253185610641408 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899420672 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚18223826 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214661 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31930 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3457276899840 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚62644230 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411624 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 _step‚248 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955590656 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 $ _timestamp‚1.788513918493908e+09 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚76330726 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177046528 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚244209858587872 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 138974954 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚249727416317952 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 139001069 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31685 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1180 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3445280711845 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚253185610641408 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31930 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚57178495 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚18224478 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3444383300221 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚253185610641408 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899420672 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚18223826 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214661 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31930 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3457276899840 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚62644230 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411624 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 _step‚248 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955590656 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 $ _timestamp‚1.788513918493908e+09 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚76330726 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177046528 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚244209858587872 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 138974954 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚249727416317952 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 139001069 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31685 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1180 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3445280711845 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚253185610641408 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31930 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 trainer/global_step‚3049 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 230944256 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 _runtime‚3543.964817728 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚247654241888093 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3458176320512 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3458176320512 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚283128922859520 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450984 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚654 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚279519875203584 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899420672 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚62644228 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917423616 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚38954669 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26115 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚76356839 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955590656 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚62644230 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚62644228 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917423616 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 138974954 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚244209875627232 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 254884352 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3608792771584 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚247655156339077 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007237 FDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450776 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26087 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚58068372 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚253629300800000 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚58069005 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚18507910 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26089 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚77565124 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚253629300800000 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚633 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚39561093 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚77539037 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3512718084096 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3511818676736 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚77565124 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 trainer/global_step‚3099 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3512718084096 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31685 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚63619230 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚253629318802944 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚257141119476736 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177046528 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26089 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚39561095 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚287562762976256 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31685 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007237 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26087 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955590656 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚248030900239504 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚253629318802944 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚257141119476736 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 141184356 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214661 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1180 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3499618707427 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚257142036887040 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚631 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3666007836160 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 % _timestamp‚1.7885139759601164e+09 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 238120448 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚248030883200144 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177046528 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚63619230 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _runtime‚3601.430604783 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚251529604496155 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 : í† G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 238120448 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚248030883200144 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177046528 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚63619230 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _runtime‚3601.430604783 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚251529604496155 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450776 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26087 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚58068372 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚253629300800000 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚58069005 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚18507910 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26089 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚77565124 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚253629300800000 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚633 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚39561093 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚77539037 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3512718084096 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3511818676736 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚77565124 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 trainer/global_step‚3099 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3512718084096 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31685 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚63619230 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚253629318802944 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚257141119476736 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177046528 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26089 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚39561095 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚287562762976256ûÉ ð~2É- _wandbruntime‚3602  _runtime‚3602ç(z,ú&ÊôM ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚253708931095552 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 _runtime‚3602.590507355 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚63639128 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚77563628 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3667388233728 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26115 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚248108880333632 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚77563628 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955590656 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31930 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3667157289472 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚251608693251601 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 141202756 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3513830900736 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚257222779999232 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3512931480064 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚63639130 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 398533632 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚77589741 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450984 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚63639128 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1205 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚39573471 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚283985836240384 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 _step‚252 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26113 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917423616 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31685 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚251609607702585 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 trainer/global_step‚3099 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007237 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3513830900736 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955590656 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚257222779999232 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3500727368953 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚257221862575616 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚18513708 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 141228871 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3499829957329 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 141228871 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚18513078 # _timestamp‚1.78851397711832e+09 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚253708949098496 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚77589741 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917423616 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 141202756 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚58086547 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214661 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 P/üZÿ&ËùM ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26115 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚248108880333632 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚77563628 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955590656 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31930 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3667157289472 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚251608693251601 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 141202756 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3513830900736 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚257222779999232 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3512931480064 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚63639130 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 398533632 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚77589741 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450984 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚63639128 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1205 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚39573471 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚283985836240384 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 _step‚252 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26113 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917423616 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31685 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚251609607702585 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 trainer/global_step‚3099 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007237 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3513830900736 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955590656 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚257222779999232 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3500727368953 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚257221862575616 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚18513708 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 141228871 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3499829957329 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 141228871 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚18513078 # _timestamp‚1.78851397711832e+09 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚253708949098496 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚77589741 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917423616 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 141202756 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚58086547 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214661 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚63639128 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚77563628 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3667388233728 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171üU[¡2Ì- _wandbruntime‚3602  _runtime‚3602 aFYÍÓ 0loss/FRt3/assignment_loss‚3.1132496165674786 &loss/total_loss‚44.004229062556654 _step‚253 trainer/global_step‚3099 2loss/SRqqt4/assignment_loss‚1.7670016512274742 /loss/FRt2/detection_loss‚1.5802973213623155 1loss/SRqqt1/detection_loss‚2.0787252224981785 .loss/FBt1/detection_loss‚2.832301232549879 .loss/FBt4/detection_loss‚3.128008450071017 _runtime‚3602.592913366 0loss/FRt2/assignment_loss‚1.3219122706719164 1loss/SRqqt4/detection_loss‚2.2836304027587175 /loss/FRt1/assignment_loss‚3.272890712533678 0loss/FBt3/assignment_loss‚0.3417315650603996 2loss/SRqqt2/assignment_loss‚1.0858356869430281 1loss/FBt2/assignment_loss‚0.09067083562831213 0loss/FBt4/assignment_loss‚0.0800047874605904 0loss/SRqqt2/detection_loss‚1.028687627389445 0loss/FRt4/assignment_loss‚3.2745907306671143 /loss/FRt3/detection_loss‚2.5297870518163195 epoch‚2 1loss/SRqqt1/assignment_loss‚1.453908371234623 2loss/SRqqt3/assignment_loss‚1.6502219516298045 0loss/FBt1/assignment_loss‚0.1901470469052179 .loss/FRt1/detection_loss‚2.101208969950676 .loss/FRt4/detection_loss‚2.679733967408538 1loss/SRqqt3/detection_loss‚1.8697523383990577 /loss/FBt2/detection_loss‚0.6526146886934494 .loss/FBt3/detection_loss‚3.597316563129425 % _timestamp‚1.7885139771225994e+097jPƒ^ÎØ .loss/FRt4/detection_loss‚2.679733967408538 1loss/SRqqt3/detection_loss‚1.8697523383990577 /loss/FBt2/detection_loss‚0.6526146886934494 .loss/FBt3/detection_loss‚3.597316563129425 % _timestamp‚1.7885139771225994e+09 0loss/FRt3/assignment_loss‚3.1132496165674786 &loss/total_loss‚44.004229062556654 _step‚253 trainer/global_step‚3099 2loss/SRqqt4/assignment_loss‚1.7670016512274742 /loss/FRt2/detection_loss‚1.5802973213623155 1loss/SRqqt1/detection_loss‚2.0787252224981785 .loss/FBt1/detection_loss‚2.832301232549879 .loss/FBt4/detection_loss‚3.128008450071017 _runtime‚3602.592913366 0loss/FRt2/assignment_loss‚1.3219122706719164 1loss/SRqqt4/detection_loss‚2.2836304027587175 /loss/FRt1/assignment_loss‚3.272890712533678 0loss/FBt3/assignment_loss‚0.3417315650603996 2loss/SRqqt2/assignment_loss‚1.0858356869430281 1loss/FBt2/assignment_loss‚0.09067083562831213 0loss/FBt4/assignment_loss‚0.0800047874605904 0loss/SRqqt2/detection_loss‚1.028687627389445 0loss/FRt4/assignment_loss‚3.2745907306671143 /loss/FRt3/detection_loss‚2.5297870518163195 epoch‚2 1loss/SRqqt1/assignment_loss‚1.453908371234623 2loss/SRqqt3/assignment_loss‚1.6502219516298045 0loss/FBt1/assignment_loss‚0.1901470469052179 .loss/FRt1/detection_loss‚2.101208969950676ý‹Ì%”Ï‚(:‰ ÅêÔ±Èæþ gpu.0.gpu‚40.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚168.715' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.17875 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0±EiLЂ(:Á ÅêÔ¼é‹ÿ disk.sdb.out‚ 412.49609375 disk.md0.out‚ 397.3984375 disk.dm-0.out‚ 40211.78125 disk.nvme0n1.in‚ 39.64453125$ proc.memory.availableMB‚649.5625* proc.memory.percent‚18.319988250732422 cpu‚25.18957482202193 network.recv‚ 1.17801e+06# disk.nvme1n1.out‚20252.78515625 disk.sda.out‚ 412.49609375 disk.sda2.in‚ 0.0390625 disk.md0.in‚ 0.0703125 disk.rbd2.in‚0 disk.sda.in‚ 0.0390625 disk.sdb1.in‚0 disk.rbd2.out‚1.0625 disk.sda1.out‚0 disk.md0p1.in‚ 0.0703125 disk.md0p1.out‚ 397.3984375 disk.sdb.in‚0.03125 disk.sdb2.out‚ 412.49609375 disk.dm-0.in‚ 230.1484375# disk.nvme0n1.out‚19958.99609375$ memory_percent‚96.03538513183594! proc.memory.rssMB‚ 3001.546875 network.sent‚ 6.983714e+06) disk./.usagePercent‚75.84477615964744% disk./.usageGB‚2712.5765686035156 disk.nvme1n1.in‚ 190.50390625 disk.sda1.in‚0 disk.sda2.out‚ 412.49609375 disk.rbd0.out‚1.125 disk.sdb1.out‚0 disk.sdb2.in‚0.03125 disk.rbd0.in‚0 proc.cpu.threads‚17Ë©¿NÄÑ‚(:‰ ÔêÔ‰»ÿ gpu.0.gpu‚55.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚37.0 gpu.0.powerWattsdzûCЂ117.525' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚29.38125 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0˜gAþNÒ‚(:à ÔêÔ£Ž²€ disk.sdb1.in‚0 network.sent‚ 7.0925e+06 disk.nvme1n1.in‚ 190.50390625 disk.md0.in‚ 0.0703125 disk.md0.out‚ 399.2265625 disk.sda1.in‚0 disk.sda2.out‚414.3916015625 disk.nvme0n1.out‚19959 disk.sdb.in‚0.03125 disk.sda.in‚ 0.0390625 disk.sda2.in‚ 0.0390625 disk.sdb2.out‚414.3916015625 disk.sda1.out‚0 disk.dm-0.out‚ 40513.9921875 disk.md0p1.out‚ 399.2265625$ memory_percent‚96.03509902954102 network.recv‚ 1.184477e+06% disk./.usageGB‚2712.5763244628906 disk.rbd2.in‚0 disk.dm-0.in‚ 230.1484375 disk.nvme0n1.in‚ 39.64453125 disk.sdb.out‚414.3916015625 cpu‚25.188914879396624 proc.cpu.threads‚17 disk.md0p1.in‚ 0.0703125 disk.sda.out‚414.3916015625 disk.sdb2.in‚0.03125! proc.memory.rssMB‚ 3001.546875 disk.rbd2.out‚1.0625& proc.memory.availableMB‚ 649.609375* proc.memory.percent‚18.319988250732422) disk./.usagePercent‚75.84476933337339 disk.rbd0.out‚1.125" disk.nvme1n1.out‚ 20554.9921875 disk.rbd0.in‚0 disk.sdb1.out‚0i(a~‘Ó‚(:† ãêÔÞÖïÿ gpu.0.gpu‚2.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚37.0 gpu.0.powerWatts‚168.992' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.248 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0r±ÊMYÔ‚(:Î ãêÔ²ë·) disk./.usagePercent‚75.84477455973946 disk.sda2.in‚ 0.0390625 disk.nvme1n1.in‚ 190.828125 disk.md0.out‚ 400.84375 disk.sda1.out‚0% disk./.usageGB‚2712.5765113830566 disk.nvme0n1.in‚ 43.7734375# disk.nvme1n1.out‚20762.83984375 disk.rbd2.out‚1.0625* proc.memory.percent‚18.319988250732422 disk.dm-0.in‚ 234.6015625 disk.rbd2.in‚0! disk.sdb2.out‚416.06787109375 network.recv‚ 1.195002e+06" disk.nvme0n1.out‚ 19959.2109375 disk.rbd0.out‚1.125 disk.md0.in‚ 0.0703125 disk.sda1.in‚0$ memory_percent‚96.03846073150635! proc.memory.rssMB‚ 3001.546875 disk.sda.in‚ 0.0390625 disk.sdb2.in‚0.03125 cpu‚25.188264384352816 proc.cpu.threads‚17 network.sent‚ 7.103667e+06 disk.md0p1.in‚ 0.0703125 disk.md0p1.out‚ 400.84375 disk.sdb.in‚0.03125 disk.sda.out‚416.06787109375! disk.sda2.out‚416.06787109375( proc.memory.availableMB‚ 649.05859375 disk.dm-0.out‚40722.05078125 disk.sdb.out‚416.06787109375 disk.rbd0.in‚0 disk.sdb1.in‚0 disk.sdb1.out‚0ÕYï¼2Õ- _wandbruntime‚3657  _runtime‚3657ŸîÅÖ¿ % lr-AdamW/pg2‚0.000792168067234838 trainer/global_step‚3149 % _timestamp‚1.7885140320086927e+09 _runtime‚3657.479036249 _step‚254 % lr-AdamW/pg1‚0.000792168067234838F¶®†Ê×Ä % lr-AdamW/pg1‚0.000792168067234838 % lr-AdamW/pg2‚0.000792168067234838 trainer/global_step‚3149 % _timestamp‚1.7885140320086927e+09 _runtime‚3657.479036249 _step‚254þW­£Â2Ø- _wandbruntime‚3657  _runtime‚36573{ )ò'ÙìO [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3568372664320 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31685 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚78798026 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26089 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚292086761171968 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚257610833580544 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚64614130 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚18797092 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚288362478056448 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26089 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚629 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚40179895 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚64614130 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚261178288834560 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177046528 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚58976358 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _step‚255 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26087 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 224666112 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177046528 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31685 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3567473256960 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1180 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 248606208 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3567473256960 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410304 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚40179893 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚78771939 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450776 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚255484055859663 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚251929887906544 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007237 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 143386069 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407360 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0%Å•þ÷'ÚñO ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚64614132 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚261178288834560 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177046528 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚58976358 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _step‚255 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26087 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 224666112 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177046528 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31685 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3567473256960 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1180 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 248606208 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3567473256960 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410304 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚40179893 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚78771939 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450776 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚255484055859663 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚251929887906544 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007237 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 143386069 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407360 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3568372664320 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31685 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚78798026 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26089 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚292086761171968 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚257610833580544 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚64614130 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚18797092 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚288362478056448 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26089 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚629 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚40179895 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚64614130 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 JDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚64634028 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚257690463876096 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚40192269 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3725445934080 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917423616 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899420672 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚257690463876096 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955590656 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3569485480960 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 143430558 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚288451797277184 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚261259949357056 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚261259949357056 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 398533632 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚40192271 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚78796530 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 224652800 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚78796530 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31685 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177046528 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214661 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 % _timestamp‚1.7885140329621124e+09 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26115 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31930 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 143456673 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚64634030 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚58994547 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3555276614437 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚3658.433140635 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917423616 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899420672 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1205 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚292177243211264 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚288451821217280 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177046528 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1180 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚612 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 aODeviceStato²fsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3725670586880 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007237 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955590656 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3568586060288 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3556174026061 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚261259031933440 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 143456673 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚64634030 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚58994547 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3555276614437 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚3658.433140635 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917423616 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899420672 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1205 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚292177243211264 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚288451821217280 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177046528 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1180 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚612 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3725670586880 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007237 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955590656 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3568586060288 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3556174026061 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚261259031933440 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚64634028 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚257690463876096 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚40192269 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3725445934080 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917423616 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899420672 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚257690463876096 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955590656 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3569485480960 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 143430558 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚288451797277184 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚261259949357056 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚261259949357056 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 398533632 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚40192271 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚78796530 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 224652800 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚78796530 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31685 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177046528 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214661 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 % _timestamp‚1.7885140329621124e+09 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26115 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31930 EDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26089 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 trainer/global_step‚3199 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚80030928 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007237 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚261592330355200 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚642 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1205 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚65609030 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚80004841 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚261592348358144 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚261592330355200 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚19087001 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚259439421673947 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26087 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚59885696 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚65609032 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚255828892612944 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚19086361 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚265215458192384 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚265215458192384 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3782737744384 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚265216375602688 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410304 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 145639960 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚259438507223171 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31685 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407360 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1180 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3609614610227 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3624027244544 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚292828439093248 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0¹ø»mÁîñO GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚255828892612944 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚19086361 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚265215458192384 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚265215458192384 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3782737744384 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚265216375602688 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410304 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 145639960 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚259438507223171 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31685 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407360 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1180 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3609614610227 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3624027244544 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚292828439093248 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚255828909652304 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26087 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3782517272576 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214661 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411416 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 _runtime‚3708.802826289 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177046528 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚261592348358144 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3623127837184 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31685 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3610512021643 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 145613871 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31930 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚65609030 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚296610956365824 % _timestamp‚1.7885140833323395e+09 _step‚259 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚65609032 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917410304 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407360 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚296611200777728 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 244411904 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26089 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚640 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 398533632 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26089 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 trainer/global_step‚3199 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚80030928 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007237 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚261592330355200 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚642 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1205 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚65609030 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚80004841 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚261592348358144 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚261592330355200 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚19087001 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚259439421673947 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26087 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚59885696 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚65609032 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0ƒ†²2ï- _wandbruntime‚3709  _runtime‚3709x~€÷ú&ðôM E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚261671960650752 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899420672 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚265297118714880 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚623 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 398533632 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚65628928 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚65628930 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917423616 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917423616 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26113 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 205778432 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26115 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177046528 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955590656 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚80029432 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3624240640512 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214661 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚59903082 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚40811069 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3783710208000 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 145684475 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31685 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 145658360 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 229718528 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚255906889746432 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚296701468521984 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚259518510429601 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450984 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31930 trainer/global_step‚3199 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26113 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚40811071 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚65628930 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚80029432 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007237 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3624240640512 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚59903707 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚625 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚352šûßÿ&ñùM E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3624240640512 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214661 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚59903082 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚40811069 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3783710208000 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 145684475 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31685 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 145658360 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 229718528 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚255906889746432 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚296701468521984 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚259518510429601 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450984 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31930 trainer/global_step‚3199 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26113 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚40811071 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚65628930 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚80029432 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007237 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3624240640512 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚59903707 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚625 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚261671960650752 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899420672 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚265297118714880 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚623 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 398533632 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚65628928 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚65628930 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917423616 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917423616 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26113 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 205778432 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26115 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177046528 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955590656 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚80029432„b=uT2ò- _wandbruntime‚3709  _runtime‚3709ëZ_úWóÑ .loss/FBt2/detection_loss‚0.524905595778959 /loss/FBt4/detection_loss‚2.8683975509234836 trainer/global_step‚3199 2loss/SRqqt1/assignment_loss‚1.3604442190378905 0loss/FBt4/assignment_loss‚0.3121223564209816 0loss/FBt1/assignment_loss‚0.2677161797570686 3loss/FBt3/assignment_loss‚0.0069718202503281645 /loss/FRt1/detection_loss‚1.9816291302455382 /loss/FRt2/detection_loss‚1.4996105842292309 -loss/FRt4/detection_loss‚2.80978069059989 .loss/FBt1/detection_loss‚2.576194362507926 %loss/total_loss‚41.76301109474757 1loss/SRqqt1/detection_loss‚1.9871739755074183 _step‚261 2loss/SRqqt2/assignment_loss‚0.9852697216585035 1loss/SRqqt3/detection_loss‚2.0093445635504192 0loss/FRt2/assignment_loss‚1.1100386748711268 2loss/SRqqt4/assignment_loss‚1.9303978173821061 .loss/FRt3/detection_loss‚2.205369849096645 1loss/SRqqt2/detection_loss‚1.0214983246854095 epoch‚2 # _timestamp‚1.78851408432085e+09 _runtime‚3709.791036609 /loss/FRt3/assignment_loss‚3.027152268453078 /loss/FRt1/assignment_loss‚3.045431349674861 0loss/FRt4/assignment_loss‚3.1949723573292004 1loss/SRqqt3/assignment_loss‚1.521886037869586 /loss/FBt3/detection_loss‚2.8293264508247375 /loss/FBt2/assignment_loss‚0.165661749677803 1loss/SRqqt4/detection_loss‚2.5217154644153736*Z.\ôÖ 0loss/FRt2/assignment_loss‚1.1100386748711268 2loss/SRqqt4/assignment_loss‚1.9303978173821061 .loss/FRt3/detection_loss‚2.205369849096645 1loss/SRqqt2/detection_loss‚1.0214983246854095 epoch‚2 # _timestamp‚1.78851408432085e+09 _runtime‚3709.791036609 /loss/FRt3/assignment_loss‚3.027152268453078 /loss/FRt1/assignment_loss‚3.045431349674861 0loss/FRt4/assignment_loss‚3.1949723573292004 1loss/SRqqt3/assignment_loss‚1.521886037869586 /loss/FBt3/detection_loss‚2.8293264508247375 /loss/FBt2/assignment_loss‚0.165661749677803 1loss/SRqqt4/detection_loss‚2.5217154644153736 .loss/FBt2/detection_loss‚0.524905595778959 /loss/FBt4/detection_loss‚2.8683975509234836 trainer/global_step‚3199 2loss/SRqqt1/assignment_loss‚1.3604442190378905 0loss/FBt4/assignment_loss‚0.3121223564209816 0loss/FBt1/assignment_loss‚0.2677161797570686 3loss/FBt3/assignment_loss‚0.0069718202503281645 /loss/FRt1/detection_loss‚1.9816291302455382 /loss/FRt2/detection_loss‚1.4996105842292309 -loss/FRt4/detection_loss‚2.80978069059989 .loss/FBt1/detection_loss‚2.576194362507926 %loss/total_loss‚41.76301109474757 1loss/SRqqt1/detection_loss‚1.9871739755074183 _step‚261 2loss/SRqqt2/assignment_loss‚0.9852697216585035 1loss/SRqqt3/detection_loss‚2.0093445635504192…2dY”õ‚(:‰ ®žêÔû§Œ€ gpu.0.gpu‚40.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚170.559' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.63975 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0âŒbÔ]ö‚(:Ò ®žêÔË̦€ disk.nvme0n1.in‚ 43.7734375 disk.sdb.in‚0.03125 disk.sdb1.out‚0 cpu‚25.184843930732395) disk./.usagePercent‚75.82371614424054% disk./.usageGB‚2711.8233604431152 disk.dm-0.in‚ 234.6015625 disk.dm-0.out‚ 41227.9765625" disk.nvme0n1.out‚ 20154.8984375 disk.sda.out‚424.54931640625 disk.sda2.in‚ 0.0390625 network.recv‚ 1.226265e+06 disk.rbd0.in‚0! disk.sdb2.out‚424.54931640625 network.sent‚ 7.251367e+06 disk.md0p1.out‚ 409.02734375 disk.md0.out‚ 409.02734375 disk.sda1.in‚0 disk.rbd0.out‚1.125 disk.nvme1n1.in‚ 190.828125 disk.sdb1.in‚0* proc.memory.percent‚18.320012092590332 disk.sda1.out‚0! disk.sda2.out‚424.54931640625 disk.rbd2.in‚0$ memory_percent‚96.04029655456543# proc.memory.rssMB‚ 3001.55078125 disk.md0p1.in‚ 0.0703125 disk.sdb.out‚424.54931640625 disk.sda.in‚ 0.0390625! disk.nvme1n1.out‚ 21073.078125 disk.md0.in‚ 0.0703125 disk.rbd2.out‚1.0625 disk.sdb2.in‚0.03125' proc.memory.availableMB‚ 648.7578125 proc.cpu.threads‚17æ\7ƒ÷‚(:ˆ ½žêÔçÓ”þ gpu.0.gpu‚47.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllo‚f¥;catedBytes‚ 14773190656 gpu.0.temp‚37.0 gpu.0.powerWatts‚116.306' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚29.0765 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚07±»óAø‚(:¶ ½žêÔ÷ªÇÿ disk.md0.in‚ 0.0703125 disk.sda1.out‚0 disk.sdb.out‚ 426.611328125# proc.memory.rssMB‚ 3001.55078125 network.recv‚ 1.230712e+06 disk.sda2.in‚ 0.0390625 disk.sda.out‚ 426.611328125 disk.sda2.out‚ 426.611328125 disk.sdb2.out‚ 426.611328125 disk.md0p1.out‚ 411.015625 disk.sdb.in‚0.03125 proc.memory.availableMB‚649" disk.nvme1n1.out‚ 21148.6015625 disk.sdb1.in‚0 disk.dm-0.in‚ 234.6015625 disk.dm-0.out‚41419.25" memory_percent‚96.038818359375* proc.memory.percent‚18.320012092590332 network.sent‚ 7.345344e+06 disk.sda.in‚ 0.0390625 disk.rbd0.in‚0 disk.nvme1n1.in‚ 190.828125 disk.sdb1.out‚0 disk.nvme0n1.in‚ 43.7734375 cpu‚25.184183575020512 proc.cpu.threads‚17 disk.rbd0.out‚1.125 disk.rbd2.in‚0 disk.rbd2.out‚1.0625 disk.sdb2.in‚0.03125" disk.nvme0n1.out‚ 20270.6484375$ disk./.usageGB‚2712.077651977539 disk.md0.out‚ 411.015625 disk.sda1.in‚0 disk.md0p1.in‚ 0.0703125( disk./.usagePercent‚75.8308262419723H>Ê”ù‚(:‰ ÌžêԆŢÿ gpu.0.gpu‚54.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚172.727' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.18175 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0yfLZú‚(:Ï ÌžêÔÿº®ÿ disk.rbd2.in‚0 disk.nvme0n1.in‚ 43.7734375 disk.sdb.out‚428.2431640625 disk.sda2.in‚ 0.0390625 disk.rbd0.out‚1.125" disk.nvme0n1.out‚ 20357.4765625 disk.sda.in‚ 0.0390625) disk./.usagePercent‚75.83780749378126 disk.sdb2.out‚428.2431640625 disk.md0p1.out‚ 412.58984375 proc.cpu.threads‚17$ disk./.usageGB‚2712.327335357666# disk.nvme1n1.out‚21198.82421875 disk.rbd2.out‚1.0625# memory_percent‚96.0392713546753* proc.memory.percent‚18.320012092590332 cpu‚25.18358691709346 disk.md0.in‚ 0.0703125 disk.md0.out‚ 412.58984375 disk.sda.out‚428.2431640625 disk.sda2.out‚428.2431640625 disk.rbd0.in‚0 network.sent‚ 7.354412e+06 disk.sdb1.in‚0( proc.memory.availableMB‚ 648.92578125# proc.memory.rssMB‚ 3001.55078125 disk.nvme1n1.in‚ 190.828125 disk.sdb1.out‚0 disk.dm-0.out‚41556.30078125 disk.md0p1.in‚ 0.0703125 network.recv‚ 1.241852e+06 disk.sdb2.in‚0.03125 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-0.in‚ 234.6015625 disk.sdb.in‚0.03125”mm2û- _wandbruntime‚3763  _runtime‚3763~ûÐÅÇüÁ _step‚262 & lr-AdamW/pg1‚0.0007854391906636837 & lr-AdamW/pg2‚0.0007854391906636837 trainer/global_step‚3249 % _timestamp‚1.7885141383502073e+09 _runtime‚3763.820604704zUPœÌýÆ & lr-AdamW/pg1‚0.0007854391906636837 & lr-AdamW/pg2‚0.0007854391906636837 trainer/global_step‚3249 % _timestamp‚1.7885141383502073e+09 _runtime‚3763.820604704 _step‚262†“ Ʋ2þ- _wandbruntime‚3763  _runtime‚3763HÕøxò'ÿìO ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3678782417408 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26089 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407360 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚60793322 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚3763.821833731 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚81237743 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31685 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚297294400130048 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3665958678751 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26087 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚81263830 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚263393873037455 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007237 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31930 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚265573863135744 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚19376442 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 238120448 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚301135267271680 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚259727914358704 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3679681824768 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚269253544960512 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31685 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚269252627550208 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚265573845132800 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚269252627550208 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚301135505392128 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚263392958586679 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚66603932 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚66603930 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚66603930 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1205 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214661 % _timestamp‚1.7885141383509853e+09 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚615 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚41417493 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3665061267335 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984?=\U÷'€ñO E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31930 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚297294424070144 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26087 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955590656 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚19375829 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚41417495 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚269253544960512 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚259727897319344 _step‚263 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚265573845132800 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 398533632 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚81237743 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3840867141632 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3841081321984 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 214180352 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚66603932 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410304 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955590656 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚265573863135744 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1180 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26089 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 trainer/global_step‚3249 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚60793937 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26089 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407360 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚60793322 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚3763.821833731 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚81237743 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31685 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚297294400130048 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3665958678751 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26087 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚81263830 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚263393873037455 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007237 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31930 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚265573863135744 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚19376442 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 238120448 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚301135267271680 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚259727914358704 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3679681824768 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚269253544960512 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31685 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚269252627550208 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚265573845132800 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚269252627550208 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚301135505392128 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚263392958586679 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚66603932 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚66603930 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚66603930 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1205 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214661 % _timestamp‚1.7885141383509853e+09 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚615 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚41417493 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3665061267335 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3678782417408 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917423616 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 398533632 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917423616 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚60811446 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚265653475428352 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚41429871 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚3764.769174383 _step‚264 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚265653493431296 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899420672 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955590656 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3680794641408 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚301225781110272 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31930 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 147912277 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚66623828 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚297383743290880 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚81262334 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450984 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚263472961793109 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚269333370649088 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 147886162 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚60812065 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚41429869 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177046528 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899420672 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1205 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚301226010828800 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚297383719350784 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚263472047342125 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚66623828 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411624 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007237 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚265653493431296 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚66623830 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚619 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚617 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214661 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3667067340277 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 trainer/global_step‚3249 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3679895220736 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1180 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 229718528!Έw'ƒûM NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚66623830 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26115 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚269334288072704 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚269334288072704 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955590656 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3680794641408 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚301225781110272 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31930 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 147912277 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚66623828 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚297383743290880 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚81262334 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450984 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚263472961793109 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚269333370649088 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 147886162 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚60812065 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚41429869 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177046528 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899420672 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1205 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚301226010828800 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚297383719350784 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚263472047342125 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚66623828 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411624 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007237 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚265653493431296 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚66623830 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚619 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚617 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214661 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3667067340277 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 trainer/global_step‚3249 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3679895220736 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1180 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 229718528 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚19382194 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚265653475428352 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 147912277 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177046528 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26113 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31685 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 147886162 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3680794641408 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚81288447 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3842061759488 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚259805894452832 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955590656 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3679895220736 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917423616 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 398533632 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917423616 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚60811446 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚265653475428352 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚41429871 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚3764.769174383 _step‚264 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚265653493431296 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899420672 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 PDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26089 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31930 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚273289796908032 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1205 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚269555377913344 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 150095564 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚629 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚273290714318336 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚82470645 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚42036295 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007237 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 _runtime‚3821.523849865 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚82470645 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 214180352 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚263626902025744 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26087 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚19664699 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3720507924443 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3734436997632 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410304 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚67598832 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚263626919065104 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955590656 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3735336404992 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31685 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚267347409950187 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚269555377913344 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚273289796908032 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3734436997632 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3721405335859 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31930 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚82496732 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26087 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚19664699 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3720507924443 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3734436997632 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410304 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚67598832 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚263626919065104 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955590656 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3735336404992 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31685 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚267347409950187 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚269555377913344 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚273289796908032 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3734436997632 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3721405335859 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31930 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚82496732 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26089 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31930 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚273289796908032 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1205 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚269555377913344 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 150095564 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚629 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚273290714318336 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚82470645 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚42036295‹sÁü¿2•- _wandbruntime‚3822  _runtime‚3822j¸ü&–öM J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31685 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 398533632 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚267426498705633 % _timestamp‚1.7885141972329493e+09 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3736449221632 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31930 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1205 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3735549800960 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214661 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚263704899159232 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚269634990205952 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚269635008208896 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚61719731 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26113 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899420672 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007237 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚67618728 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177046528 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚301849680387584 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚301849704327680 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚269635008208896 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚269634990205952 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3722513997385 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚42048669 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚67618730 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955590656 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚263704882119872 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955590656 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚61719112 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 209972736 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917423616 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411624 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26115 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 150113964 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚67618728 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1180 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 _step‚268 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3736449221632 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 150113964 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450984 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26113 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3900214348800 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚3299 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚273371457430528 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3900424321536 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 150140079 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 233912832 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚82521349 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚619 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚305750128649216 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899420672 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚42048671¡C'—ûM J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31685 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 398533632 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚267426498705633 % _timestamp‚1.7885141972329493e+09 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3736449221632 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31930 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1205 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3735549800960 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214661 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚263704899159232 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚269634990205952 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚269635008208896 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚61719731 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26113 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899420672 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007237 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚67618728 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177046528 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚301849680387584 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚301849704327680 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚269635008208896 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚269634990205952 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3722513997385 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚42048669 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚67618730 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955590656 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚263704882119872 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955590656 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚61719112 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 209972736 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917423616 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411624 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26115 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 150113964 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚67618728 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1180 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 _step‚268 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3736449221632 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 150113964 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450984 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26113 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3900214348800 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚3299 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚273371457430528 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3900424321536 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 150140079 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 233912832 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚82521349 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚619 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚305750128649216 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899420672 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚42048671Œñ5°Á2˜- _wandbruntime‚3822  _runtime‚3822Ç\¦W™Ñ 0loss/FRt2/assignment_loss‚1.3017001539468764 0loss/FBt2/assignment_loss‚0.1140000502870777 _step‚269 0loss/FRt1/assignment_loss‚3.3627044327405033 0loss/FBt3/assignment_loss‚0.7094373544290041 1loss/SRqqt4/detection_loss‚2.4999279349278183 .loss/FBt1/detection_loss‚2.920775920152664 /loss/FRt2/detection_loss‚1.7624098677188158 3loss/FBt4/assignment_loss‚0.0020818215012354813 /loss/FRt1/detection_loss‚2.1851114241444334 1loss/SRqqt3/detection_loss‚1.7194731125030025 _runtime‚3822.707749698 /loss/FRt3/assignment_loss‚2.899587688346704 1loss/SRqqt2/assignment_loss‚1.011380503818925 0loss/FBt1/assignment_loss‚0.5332840712607094 1loss/SRqqt1/detection_loss‚1.6665316854871937 .loss/FBt3/detection_loss‚2.584160010019938 /loss/FBt4/detection_loss‚3.0373687893152237 /loss/FRt4/assignment_loss‚2.485848649551994 1loss/SRqqt3/assignment_loss‚1.219300045535482 0loss/SRqqt2/detection_loss‚1.093183493840375 .loss/FRt4/detection_loss‚3.467886303600512 /loss/FBt2/detection_loss‚0.7467355319623197 $loss/total_loss‚43.1916111263772 epoch‚2 1loss/SRqqt1/assignment_loss‚1.796117188061698 2loss/SRqqt4/assignment_loss‚1.7205463609634302 /loss/FRt3/detection_loss‚2.3520587322612605 trainer/global_step‚3299 $ _timestamp‚1.788514197237422e+09ä¨,|\šÖ .loss/FRt4/detection_loss‚3.467886303600512 /loss/FBt2/detection_loss‚0.7467355319623197 $loss/total_loss‚43.1916111263772 epoch‚2 1loss/SRqqt1/assignment_loss‚1.796117188061698 2loss/SRqqt4/assignment_loss‚1.7205463609634302 /loss/FRt3/detection_loss‚2.3520587322612605 trainer/global_step‚3299 $ _timestamp‚1.788514197237422e+09 0loss/FRt2/assignment_loss‚1.3017001539468764 0loss/FBt2/assignment_loss‚0.1140000502870777 _step‚269 0loss/FRt1/assignment_loss‚3.3627044327405033 0loss/FBt3/assignment_loss‚0.7094373544290041 1loss/SRqqt4/detection_loss‚2.4999279349278183 .loss/FBt1/detection_loss‚2.920775920152664 /loss/FRt2/detection_loss‚1.7624098677188158 3loss/FBt4/assignment_loss‚0.0020818215012354813 /loss/FRt1/detection_loss‚2.1851114241444334 1loss/SRqqt3/detection_loss‚1.7194731125030025 _runtime‚3822.707749698 /loss/FRt3/assignment_loss‚2.899587688346704 1loss/SRqqt2/assignment_loss‚1.011380503818925 0loss/FBt1/assignment_loss‚0.5332840712607094 1loss/SRqqt1/detection_loss‚1.6665316854871937 .loss/FBt3/detection_loss‚2.584160010019938 /loss/FBt4/detection_loss‚3.0373687893152237 /loss/FRt4/assignment_loss‚2.485848649551994 1loss/SRqqt3/assignment_loss‚1.219300045535482 0loss/SRqqt2/detection_loss‚1.093183493840375P @Ü”›‚(:‰ —ŸêÔªŸ‰€ gpu.0.gpu‚40.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚172.727' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.18175 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0èGÝYœ‚(:Î —ŸêÔϞƀ( proc.memory.availableMB‚ 648.99609375 network.sent‚ 7.500356e+06" disk.nvme0n1.out‚ 20667.1484375 disk.sda2.in‚ 0.0390625 disk.rbd2.in‚0 proc.cpu.threads‚17 disk.sdb2.in‚0.03125 disk.sdb.in‚0.03125 disk.rbd2.out‚1.0625 disk.sda1.in‚0 disk.nvme1n1.in‚ 190.828125 network.recv‚ 1.295044e+06 disk.md0.out‚ 420.703125 disk.sda1.out‚0 disk.md0p1.out‚ 420.703125$ memory_percent‚96.03884220123291# proc.memory.rssMB‚ 3001.55078125 disk.dm-0.out‚ 42694.4921875% disk./.usageGB‚2712.0744018554688 disk.nvme0n1.in‚ 43.7734375! disk.sda2.out‚436.69384765625 disk.rbd0.in‚0 disk.nvme1n1.out‚ 22027.34375 disk.sdb1.in‚0! disk.sdb2.out‚436.69384765625) disk./.usagePercent‚75.83073536719893 disk.sda.in‚ 0.0390625 disk.sda.out‚436.69384765625 disk.sdb1.out‚0 disk.dm-0.in‚ 234.6015625 disk.md0p1.in‚ 0.0703125* proc.memory.percent‚18.320012092590332 cpu‚25.179605104020116 disk.sdb.out‚436.69384765625 disk.rbd0.out‚1.125 disk.md0.in‚ 0.0703125ÉÅ;@6‚(:‰ ¦ŸêÔ•Þ±ÿ gpu.0.gpu‚55.0 gpu.0¸.Uº^.memory‚7, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚172.727' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.18175 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Á*?KTž‚(:É ¦ŸêÔÙôíÿ disk.sdb.in‚0.03125 cpu‚25.17891753755661 disk.sdb1.in‚0 disk.nvme0n1.out‚ 20667.15625 disk.sda.out‚438.76611328125 disk.nvme1n1.in‚ 190.828125 proc.cpu.threads‚17 disk.md0p1.in‚ 0.0703125 disk.nvme0n1.in‚ 43.7734375 disk.rbd0.in‚0 disk.sdb1.out‚0 disk.md0.in‚ 0.0703125 disk.rbd2.in‚0& proc.memory.availableMB‚ 649.015625 disk.dm-0.in‚ 234.6015625 disk.sda2.in‚ 0.0390625 disk.md0.out‚ 422.71875$ memory_percent‚96.03872299194336 disk.sda1.in‚0# disk.nvme1n1.out‚22058.79296875( disk./.usagePercent‚75.8316749398264 disk.sdb2.in‚0.03125! disk.sdb2.out‚438.76611328125 disk.md0p1.out‚ 422.71875 disk.sdb.out‚438.76611328125 disk.rbd2.out‚1.0625 network.sent‚ 7.608334e+06 network.recv‚ 1.30065e+06 disk.dm-0.out‚42725.94921875 disk.sda.in‚ 0.0390625 disk.rbd0.out‚1.125! disk.sda2.out‚438.76611328125# proc.memory.rssMB‚ 3001.55078125* proc.memory.percent‚18.320012092590332% disk./.usageGB‚2712.1080055236816 disk.sda1.out‚0ªê6U’Ÿ‚(:‡ µŸêÔ¡Óÿ gpu.0.gpu‚56.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚37.0 gpu.0.powerWatts‚116.916' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚29.229 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0½:Ù¡O ‚(:Ä µŸêÔŸ¤Ž€) disk./.usagePercent‚75.82228028015712 disk.dm-0.in‚ 234.6015625 disk.md0p1.out‚ 424.15625 disk.sda.out‚ 440.267578125 disk.rbd2.in‚0( proc.memory.availableMB‚ 648.98828125 disk.sdb.out‚ 440.267578125 disk.sda.in‚ 0.0390625 disk.rbd0.out‚1.125 disk.dm-0.out‚ 42934.1328125 disk.nvme0n1.in‚ 43.7734375 disk.sda2.in‚ 0.0390625 disk.sda2.out‚ 440.267578125 disk.sdb1.in‚0 disk.md0p1.in‚ 0.0703125" disk.nvme0n1.out‚ 20667.3515625 disk.nvme1n1.out‚ 22266.78125 disk.sdb1.out‚0 disk.sdb2.out‚ 440.267578125 disk.rbd2.out‚1.0625 cpu‚25.178286951066237 network.sent‚ 7.615799e+06% disk./.usageGB‚2711.7720069885254 disk.rbd0.in‚0 disk.md0.in‚ 0.0703125 disk.md0.out‚ 424.15625 disk.sdb2.in‚0.03125 disk.sda1.out‚0$ memory_percent‚96.03888988494873 disk.sdb.in‚0.03125 disk.nvme1n1.in‚ 190.828125 disk.sda1.in‚0* proc.memory.percent‚18.320012092590332 network.recv‚ 1.304259e+06# proc.memory.rssMB‚ 3001.55078125 proc.cpu.threads‚17:Ô††“¡‚(:ˆ ÄŸêÔ­ž°ÿ gpu.0.gpu‚39.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚39.0 gpu.0.powerWatts‚167.774' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚41.9435 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ý}¹°T¢‚(:É ÄŸêÔÒˆ“€ disk.sda1.in‚0 disk.md0p1.in‚ 0.0703125# disk.nvme0n1.out‚20667.35546875 disk.sda.out‚ 441.775390625 disk.dm-0.out‚ 43066.5234375# proc.memory.rssMB‚ 3001.55078125 disk.sdb2.in‚0.03125* proc.memory.percent‚18.320012092590332 disk.sda2.out‚ 441.775390625) disk./.usagePercent‚75.82927379792727 disk.sda.in‚ 0.0390625 disk.rbd0.out‚1.125 disk.sdb1.in‚0 disk.md0.in‚ 0.0703125$ memory_percent‚96.03891372680664 proc.cpu.threads‚17 network.recv‚ 1.308657e+06 disk.sdb.out‚ 441.775390625 disk.rbd2.out‚1.0625 disk.sda1.out‚0 disk.sda2.in‚ 0.0390625 disk.md0p1.out‚ 425.6015625 disk.nvme0n1.in‚ 43.7734375 disk.rbd0.in‚0 disk.nvme1n1.in‚ 190.828125$ disk./.usageGB‚2712.022129058838 disk.sdb.in‚0.03125# disk.nvme1n1.out‚22399.16796875 disk.dm-0.in‚ 234.6015625 network.sent‚ 7.621773e+06 disk.sdb1.out‚0 disk.md0.out‚ 425.6015625 disk.sdb2.out‚ 441.775390625& proc.memory.availableMB‚ 648.984375 disk.rbd2.in‚0 cpu‚25.177663994195946 ×2£- _wandbruntime‚3875  _runtime‚3875æk~ǤÁ _runtime‚3875.333449372 _step‚270 & lr-AdamW/pg1‚0.0007784130792179463 & lr-AdamW/pg2‚0.0007784130792179463 trainer/global_step‚3349 % _timestamp‚1.7885142498630884e+09Ç’îÌ¥Æ & lr-AdamW/pg1‚0.0007784130792179463 & lr-AdamW/pg2‚0.0007784130792179463 trainer/global_step‚3349 % _timestamp‚1.7885142498630884e+09 _runtime‚3875.333449372 _step‚270Ž‘” 2¦- _wandbruntime‚3875  _runtime‚3875×ÆÄŒñ'§ëO ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3790990985216 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚68593732 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚68593730 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚310183626596864 $ _timestamp‚1.788514249863848e+09 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚273536892690944 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚68593730 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚271302775764471 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚19953579 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31685 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214661 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3790091577856 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26087 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1187 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚277327883676160 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚42655093 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26089 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26087 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407360 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1212 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 398533632 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 214180352 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450776 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚277326966265856 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚62608073 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚83729634 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _step‚271 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955590656 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917410304 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚19952980 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3776851992967 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31930 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3957280453120 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚271301861313695 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚310183388476416 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411416 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚273536874688000 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 152323366 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31685 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26089 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚62608674 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 trainer/global_step‚3349 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007237 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚83703547 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177046528 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3790091577856 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚601 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚267525906732144 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚277326966265856 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚68593732 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚599 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚277327883676160 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0ækÖ[ö'¨ðO R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚277326966265856 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚68593732 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚599 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚277327883676160 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3790990985216 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚68593732 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚68593730 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚310183626596864 $ _timestamp‚1.788514249863848e+09 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚273536892690944 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚68593730 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚271302775764471 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚19953579 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31685 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214661 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3790091577856 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26087 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1187 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚277327883676160 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚42655093 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26089 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26087 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407360 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1212 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 398533632 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 214180352 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450776 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚277326966265856 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚62608073 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚83729634 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _step‚271 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955590656 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917410304 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚19952980 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3776851992967 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31930 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3957280453120 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚271301861313695 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚310183388476416 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411416 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚273536874688000 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 152323366 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31685 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26089 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚62608674 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 trainer/global_step‚3349 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007237 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚83703547 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177046528 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3790091577856 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚601 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚267525906732144 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 JDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 152367881 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 _step‚272 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 244398592 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3958233019392 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31930 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚273616522986496 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3791204381184 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚310274118842368 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917423616 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955590656 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31930 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚83754251 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214661 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚267603903865632 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31685 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450984 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3792103801856 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚68613628 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚271381864520125 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899420672 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚277407709364736 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26113 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177046528 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚42667471 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3792103801856 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚68613630 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚42667469 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚267603886826272 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚62626717 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26113 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚271380950069141 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411624 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚273616522986496 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚19959246 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚68613630 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚19958642 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 220458496 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 152341766 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚83728138 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3777960654493 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 % _timestamp‚1.7885142508275387e+09 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 152367881 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚83754251 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3791204381184 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_splöTÖQè it_bytes.small_pool.peak‚ 361446400 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚62626111 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚310273874443776 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚83728138 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚604 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1187 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3958453477888 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3777063242869 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007237 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26115 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚68613628 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚273616504983552 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899420672 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚277408626788352 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0âi]h'«ûM E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚62626111 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚310273874443776 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚83728138 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚604 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1187 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3958453477888 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3777063242869 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007237 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26115 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚68613628 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚273616504983552 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899420672 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚277408626788352 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955590656 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚277408626788352 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 152367881 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 _step‚272 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 244398592 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3958233019392 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31930 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚273616522986496 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3791204381184 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚310274118842368 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917423616 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955590656 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31930 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚83754251 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214661 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚267603903865632 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31685 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450984 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3792103801856 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚68613628 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚271381864520125 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899420672 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚277407709364736 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26113 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177046528 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚42667471 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3792103801856 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚68613630 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚42667469 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚267603886826272 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚62626717 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26113 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚271380950069141 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411624 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚273616522986496 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚19959246 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚68613630 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚19958642 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 220458496 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 152341766 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚83728138 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3777960654493 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 % _timestamp‚1.7885142508275387e+09 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 152367881 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚83754251 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3791204381184 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26089 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚84936449 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚20242090 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31930 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007237 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31685 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 240217600 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 216277504 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3832298650075 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26087 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚625 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26089 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407360 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1187 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4015408713728 _runtime‚3929.748262848 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3846645565440 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚314707691954176 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3831401238659 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚281364135623680 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407360 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚281364135623680 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 154551168 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411416 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚277518407468544 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚43273893 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚623 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚84962536 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450776 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚43273895 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚277518389465600 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚281365053033984 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚63515983 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚314707932171776 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚277518407468544 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚310692283240448 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0”D¸ ºñO ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3846645565440 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚314707691954176 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3831401238659 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚281364135623680 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407360 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚281364135623680 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 154551168 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411416 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚277518407468544 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚43273893 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚623 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚84962536 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450776 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚43273895 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚277518389465600 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚281365053033984 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚63515983 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚314707932171776 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚277518407468544 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚310692283240448 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚69588632 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3845746158080 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚84962536 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 398533632 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚271424928477904 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚69588630 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26089 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚84936449 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚20242090 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31930 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007237 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31685 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 240217600 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 216277504 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3832298650075 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26087 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚625 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26089 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407360 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1187 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4015408713728 _runtime‚3929.748262848“)”Jq”»‚(:‰ € êÔª©Äþ gpu.0.gpu‚33.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚177.071' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚44.26775 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0–í5Y¼‚(:Î € êÔ˜‘üþ disk.sda2.in‚ 0.0390625 network.sent‚ 7.744784e+06) disk./.usagePercent‚75.81606101118963$ disk./.usageGB‚2711.549575805664 disk.rbd0.in‚0 disk.rbd2.out‚1.0625 disk.sda2.out‚448.9189453125 cpu‚25.17504791634382 disk.sdb2.out‚448.9189453125 disk.sda1.out‚0 disk.dm-0.out‚43743.65234375 disk.md0.in‚ 0.0703125 disk.sdb.out‚448.9189453125 disk.nvme1n1.in‚ 190.828125 disk.md0p1.out‚ 432.49609375 disk.nvme0n1.in‚ 43.7734375& proc.memory.availableMB‚ 733.234375 proc.cpu.threads‚17 network.recv‚ 1.326881e+06 disk.rbd0.out‚1.125# disk.nvme1n1.out‚23076.06640625 disk.rbd2.in‚0 disk.sda1.in‚0 disk.sdb.in‚0.03125 disk.sdb1.out‚0 disk.sdb2.in‚0.03125 disk.sda.in‚ 0.0390625 disk.sda.out‚448.9189453125* proc.memory.percent‚17.775559425354004 disk.sdb1.in‚0 disk.dm-0.in‚ 234.6015625$ memory_percent‚95.52469253540039# proc.memory.rssMB‚ 2912.34765625 disk.md0.out‚ 432.49609375 disk.md0p1.in‚ 0.0703125" disk.nvme0n1.out‚ 20667.5859375^¹þ@2½- _wandbruntime‚3930  _runtime‚3930^Vdü&¾öM NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚84987153 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚84987153 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚63534162 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3833407311601 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚84961040 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚69608530 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚599 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214661 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚271502908572032 _step‚276 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3847758382080 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31685 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚310781602461184 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26115 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚281445796146176 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 154569568 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31930 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26113 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3832509899977 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955590656 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚281445796146176 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917423616 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3847758382080 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 154569568 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚43286269 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177046528 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4016555348992 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 154595683 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚02å'¿ûM H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚601 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3846858961408 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 254884352 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450984 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚63534162 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3833407311601 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚84961040 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚69608530 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚599 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214661 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚271502908572032 _step‚276 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3847758382080 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31685 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚310781602461184 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26115 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚281445796146176 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 154569568 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31930 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26113 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3832509899977 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955590656 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚281445796146176 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917423616 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3847758382080 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 154569568 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚43286269 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177046528 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4016555348992 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 154595683 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚84987153 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚84987153 FDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214661 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31930 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚70583532 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26089 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚281499922246144 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3901400738304 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚20532529 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4073960568832 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31930 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955590656 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚70583532 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31685 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31685 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3887745307183 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚86169351 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007237 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚285401304981504 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚70583530 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚20531908 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚621 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚275323916144944 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712’¥ ¯ö'ÎðO ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚20531908 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚621 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚275323916144944 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177046528 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚279210764040711 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917410304 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚319232447160832 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚275323933184304 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411416 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1187 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚315158268217344 _runtime‚3987.351666315 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26089 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3902300145664 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚70583530 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1212 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214661 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31930 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚70583532 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26089 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚281499922246144 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3901400738304 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚20532529 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4073960568832 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31930 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955590656 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚70583532 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31685 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31685 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3887745307183 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚86169351 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007237 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚285401304981504 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚70583530—ôúÖ2Ï- _wandbruntime‚3988  _runtime‚3988£²bsÐöM [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚281579552541696 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚285482965504000 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚281579534538752 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955590656 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚70603430 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚64443326 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚64442696 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚279290767247141 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31930 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31930 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚20538255 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚70603430 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214661 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 252787200 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917423616 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 398533632 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 156823485 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31685 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚315247587438080 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955590656 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899420672 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚86193942 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚275401896239072 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3887956557085 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚86193942 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177046528 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚281579552541696 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 228847104 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 _step‚280 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚70603428 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚628 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚319322930271232 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚315247563497984 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4075342833152 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007237 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3902513541632 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚285482048080384 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚43905069 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚279289852796157 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚86220055 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3903412962304 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚281579534538752 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3902513541632 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚86220055 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411624 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 156823¬­Òª‰ 485 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1187 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917423616 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177046528 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚285482965504000 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚281579534538752 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955590656 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚70603430 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚64443326 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚64442696 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚279290767247141 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31930 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31930 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚20538255 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚70603430 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214661 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 252787200 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917423616 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 398533632 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 156823485 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31685 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚315247587438080 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955590656 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899420672 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚86193942 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚275401896239072 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3887956557085 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚86193942 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177046528 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚281579552541696 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 228847104 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 _step‚280 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚70603428 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚628 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚319322930271232 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚315247563497984 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4075342833152 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007237 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3902513541632 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚285482048080384 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚43905069 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚279289852796157 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚86220055 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3903412962304 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚281579534538752 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3902513541632 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚86220055 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411624 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 156823485 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1187 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917423616 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177046528 FDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31685 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚319624205314048 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411416 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3957954725888 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚71578430 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26089 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚65334930 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚65334305 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚87402253 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3957055318528 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177046528 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚279222937890704 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410304 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 159006772 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4132334793728 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407360 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚87402253 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚87428340 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚44511495 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917410304 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚289438474339328 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26089 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955590656 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1187 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚283166129854995 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450776 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚87428340 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407360 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚285481419020800 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26087 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _step‚283æŒ3KâðO TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.pÜ's2«'eak‚ 12835443712 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26089 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955590656 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1187 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚283166129854995 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450776 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚87428340 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407360 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚285481419020800 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26087 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _step‚283 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚20823435 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚20822812 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚71578432 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3957055318528 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31930 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚285481437023744 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚289438474339328 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3957954725888 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚44511493 trainer/global_step‚3499 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31930 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 238120448 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31685 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚319624205314048 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411416 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3957954725888 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚71578430 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26089 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚65334930 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚65334305 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚87402253 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3957055318528 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177046528 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚279222937890704 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410304 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 159006772 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4132334793728 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407360 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚87402253 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚87428340 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚44511495 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917410304 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚289438474339328›ÅlÊâ2ã- _wandbruntime‚4046  _runtime‚4046-°èDü&äöM FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚4046.624261994 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚87426844 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚634 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚279300900945472 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007237 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955590656 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3959067542528 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31930 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚632 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚289520134861824 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚87452957 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899420672 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177046528 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚323846806361088 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚87426844 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955590656 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚20829164 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214661 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3959067542528 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899420672 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚71598328 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚87452957 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31930 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚285561049316352 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 159051287 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚65353035 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚285561067319296 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚319713548474880 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚284 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚65352401 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚44523869 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚285561049316352 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26113 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 226749952 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚289519217438208 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917423616 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 250690048 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411624 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 trainer/global_step‚3499 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26115 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚71598328 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4133508576256 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚20828532 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3944300625817 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3958168121856 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚71598330 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3958168121856 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1187 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚319713524534784 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450984 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 159051287 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 159025172?„ö`'åûM WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚285561049316352 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26113 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 226749952 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚289519217438208 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917423616 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 250690048 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411624 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 trainer/global_step‚3499 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26115 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚71598328 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4133508576256 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚20828532 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3944300625817 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3958168121856 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚71598330 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3958168121856 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1187 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚319713524534784 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450984 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 159051287 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 159025172 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚4046.624261994 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚87426844 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚634 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚279300900945472 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007237 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955590656 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3959067542528 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31930 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚632 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚289520134861824 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚87452957 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899420672 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177046528 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚323846806361088 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚87426844 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955590656 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚20829164 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214661 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3959067542528 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899420672 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚71598328 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚87452957 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31930 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚285561049316352 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 159051287 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚65353035 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚285561067319296 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚319713548474880 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚284 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚65352401 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚44523869 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171œYõa=2æ- _wandbruntime‚4046  _runtime‚4046†Áè[çÕ /loss/FRt2/assignment_loss‚1.220793065699664 0loss/FRt4/assignment_loss‚3.0084661662578585 2loss/SRqqt2/assignment_loss‚0.7613586939871311 1loss/SRqqt4/detection_loss‚2.2189036223196212 .loss/FBt1/detection_loss‚2.403129730373621 /loss/FBt2/detection_loss‚0.5957294599890398 %loss/total_loss‚42.27005459836178 /loss/FBt4/detection_loss‚3.0028718284198215 /loss/FRt1/assignment_loss‚2.961569145017741 0loss/FRt3/assignment_loss‚2.2847645822912455 .loss/FRt2/detection_loss‚1.763778037137606 2loss/SRqqt1/assignment_loss‚1.6753812561011208 0loss/SRqqt2/detection_loss‚1.183891224550704 epoch‚2 1loss/FBt3/assignment_loss‚0.11561976775660046 _runtime‚4046.626309123 /loss/FRt1/detection_loss‚2.2077972427096655 trainer/global_step‚3499 2loss/FBt1/assignment_loss‚0.033114723885944386 -loss/FRt3/detection_loss‚2.72779498482123 .loss/FRt4/detection_loss‚3.641635800898075 0loss/SRqqt1/detection_loss‚1.670573744656784 1loss/SRqqt3/detection_loss‚1.8057020636750203 2loss/SRqqt3/assignment_loss‚1.5242982808336176 2loss/SRqqt4/assignment_loss‚2.3007946553970537 _step‚285 0loss/FBt2/assignment_loss‚0.1586914847384736 2loss/FBt4/assignment_loss‚0.057462640023524206 /loss/FBt3/detection_loss‚2.9459323968206133 % _timestamp‚1.7885144211561906e+09°¦ThèÚ .loss/FBt1/detection_loss‚2.403129730373621 /loss/FBt2/detection_loss‚0.5957294599890398 %loss/total_loss‚42.27005459836178 /loss/FBt4/detection_loss‚3.0028718284198215 /loss/FRt1/assignment_loss‚2.961569145017741 0loss/FRt3/assignment_loss‚2.2847645822912455 .loss/FRt2/detection_loss‚1.763778037137606 2loss/SRqqt1/assignment_loss‚1.6753812561011208 0loss/SRqqt2/detection_loss‚1.183891224550704 epoch‚2 1loss/FBt3/assignment_loss‚0.11561976775660046 _runtime‚4046.626309123 /loss/FRt1/detection_loss‚2.2077972427096655 trainer/global_step‚3499 2loss/FBt1/assignment_loss‚0.033114723885944386 -loss/FRt3/detection_loss‚2.72779498482123 .loss/FRt4/detection_loss‚3.641635800898075 0loss/SRqqt1/detection_loss‚1.670573744656784 1loss/SRqqt3/detection_loss‚1.8057020636750203 2loss/SRqqt3/assignment_loss‚1.5242982808336176 2loss/SRqqt4/assignment_loss‚2.3007946553970537 _step‚285 0loss/FBt2/assignment_loss‚0.1586914847384736 2loss/FBt4/assignment_loss‚0.057462640023524206 /loss/FBt3/detection_loss‚2.94593239682061Ìpö½ø33 % _timestamp‚1.7885144211561906e+09 /loss/FRt2/assignment_loss‚1.220793065699664 0loss/FRt4/assignment_loss‚3.0084661662578585 2loss/SRqqt2/assignment_loss‚0.7613586939871311 1loss/SRqqt4/detection_loss‚2.2189036223196212úôpxbé‚(:× ø êÔݵ* proc.memory.percent‚17.775559425354004 disk.md0.out‚ 448.94921875 disk.rbd2.in‚0 disk.dm-0.in‚ 445.58203125 disk.sdb.in‚ 0.08203125 disk.sda.in‚0.09375 proc.cpu.threads‚17 disk.dm-0.out‚45595.11328125 disk.sdb.out‚465.9033203125 disk.sda2.out‚465.9033203125) disk./.usagePercent‚75.81863654305877" disk.nvme1n1.out‚ 24279.6953125$ memory_percent‚95.52733898162842 network.recv‚ 1.363049e+06$ disk./.usageGB‚2711.641689300537 disk.md0.in‚ 0.17578125 disk.sdb2.in‚ 0.08203125 disk.md0p1.in‚ 0.17578125 disk.rbd0.out‚1.125# proc.memory.rssMB‚ 2912.34765625 disk.sda1.in‚0 disk.md0p1.out‚ 448.94921875 disk.sda.out‚465.9033203125 network.sent‚ 8.074401e+06 disk.rbd0.in‚0 disk.nvme1n1.in‚ 321.30078125 disk.sda1.out‚0 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.rbd2.out‚1.0625 disk.sdb2.out‚465.9033203125 disk.nvme0n1.in‚ 124.28125 cpu‚25.169683182783224# disk.nvme0n1.out‚21315.41796875 disk.sda2.in‚0.09375( proc.memory.availableMB‚ 732.80078125>íoâ‘ê‚(:† ø êÔè‹®€ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚173.344' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.336 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ÿ$ñÃ`ë‚(:Õ ‡¡êÔöœƒÿ# disk.nvme0n1.out‚21383.97265625! disk.sda2.out‚467.99658203125 disk.rbd0.out‚1.125 disk.md0p1.in‚ 0.17578125 network.sent‚ 8.184154e+06 network.recv‚ 1.367956e+06 disk.sda1.in‚0 disk.dm-0.in‚ 445.58203125 disk.sdb.in‚ 0.08203125 disk.rbd2.in‚0 disk.sda.in‚0.09375 disk.sda2.in‚0.09375 disk.rbd0.in‚0 cpu‚25.16908670181441 proc.cpu.threads‚17) disk./.usagePercent‚75.80485824217982$ disk./.usageGB‚2711.148910522461 disk.md0p1.out‚ 450.98828125 disk.nvme0n1.in‚ 124.28125 disk.sda.out‚467.99658203125 disk.md0.in‚ 0.17578125 disk.dm-0.out‚ 45748.21875 disk.sdb.out‚467.99658203125 disk.sdb1.in‚0 disk.md0.out‚ 450.98828125! disk.sdb2.out‚467.99658203125 disk.nvme1n1.in‚ 321.30078125# disk.nvme1n1.out‚24364.24609375$ memory_percent‚95.52745819091797% proc.memory.availableMB‚ 732.78125* proc.memory.percent‚17.775559425354004 disk.rbd2.out‚1.0625 disk.sda1.out‚0# proc.memory.rssMB‚ 2912.34765625 disk.sdb1.out‚0 disk.sdb2.in‚ 0.08203125F‚s'‘ì‚(:† ‡¡êÔƒºÉÿ gpu.0.gpu‚9.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚39.0 gpu.0.powerWatts‚173.344' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.336 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‘C•([í‚(:Ð –¡êÔñ…Æÿ disk.rbd0.out‚1.125 disk.sdb1.in‚0# proc.memory.rssMB‚ 2912.34765625* proc.memory.percent‚17.775559425354004 disk.rbd2.in‚0 disk.sda1.out‚0 disk.dm-0.out‚45748.40234375 disk.sdb.in‚ 0.08203125 disk.sdb1.out‚0 disk.md0.out‚ 452.53125 disk.sdb2.out‚469.6005859375 disk.md0p1.in‚ 0.17578125( proc.memory.availableMB‚ 732.53515625 network.sent‚ 8.18833e+06 disk.rbd0.in‚0 disk.md0.in‚ 0.17578125 disk.md0p1.out‚ 452.53125 disk.nvme0n1.in‚ 124.28125 disk.sda.in‚0.09375" disk.nvme1n1.out‚ 24364.4140625 disk.sda1.in‚0# disk.nvme0n1.out‚21383.98828125$ memory_percent‚95.52896022796631 disk.sda2.in‚0.09375 disk.nvme1n1.in‚ 321.30078125 cpu‚25.16852307571466 network.recv‚ 1.372652e+06) disk./.usagePercent‚75.80619405868411% disk./.usageGB‚2711.1966857910156 disk.sda.out‚469.6005859375 disk.sdb2.in‚ 0.08203125 disk.dm-0.in‚ 445.58203125 disk.sdb.out‚469.6005859375 proc.cpu.threads‚17 disk.sda2.out‚469.6005859375 disk.rbd2.out‚1.0625Lh Í”î‚(:‰ –¡êÔ±çªÿ gpu.0.gpu‚55.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚133.349' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚33.33725 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‰hE”ï‚(:‰ ¥¡êÔ—“üý gpu.0.gpu‚51.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚164.649' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚41.16225 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0F&fÝað‚(:Ö ¥¡êÔ‰Ò¬ÿ disk.dm-0.out‚45799.93359375 disk.md0p1.out‚ 454.15625 disk.nvme0n1.in‚ 124.28125 disk.sdb.out‚471.29052734375 disk.sda2.in‚0.09375 disk.rbd0.out‚1.125# disk.nvme1n1.out‚24366.30078125 disk.sdb1.in‚0 network.sent‚ 8.195755e+06 disk.sda.out‚471.29052734375 disk.md0.in‚ 0.17578125 cpu‚25.167972702076185% disk./.usageGB‚2711.1605834960938 disk.nvme1n1.in‚ 321.30078125 disk.sdb1.out‚0 disk.rbd2.in‚0! disk.sdb2.out‚471.29052734375 network.recv‚ 1.375951e+06 disk.rbd2.out‚1.0625( proc.memory.availableMB‚ 732.71484375* proc.memory.percent‚17.775559425354004 disk.sda1.in‚0 disk.sda1.out‚0 disk.rbd0.in‚0 proc.cpu.threads‚17) disk./.usagePercent‚75.80518462340812 disk.sda.in‚0.09375 disk.sdb2.in‚ 0.08203125" disk.nvme0n1.out‚ 21433.6328125! disk.sda2.out‚471.29052734375 disk.md0.out‚ 454.15625$ memory_percent‚95.52786350250244# proc.memory.rssMB‚ 2912.34765625 disk.dm-0.in‚ 445.58203125 disk.md0p1.in‚ 0.17578125 disk.sdb.in‚ 0.08203125«°2ñ- _wandbruntime‚4097  _runtime‚40972 $õÇòÁ _runtime‚4097.515514468 _step‚286 & lr-AdamW/pg1‚0.0007634928210581681 & lr-AdamW/pg2‚0.0007634928210581681 trainer/global_step‚3549 % _timestamp‚1.7885144720452056e+09|®¯$ÌóÆ trainer/global_step‚3549 % _timestamp‚1.7885144720452056e+09 _runtime‚4097.515514468 _step‚286 & lr-AdamW/pg1‚0.0007634928210581681 & lr-AdamW/pg2‚0.0007634928210581681žŠo2ô- _wandbruntime‚4097  _runtime‚4097G´ZUò'õìO A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007237 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 161208485 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚289462933798400 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4013609306112 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚66242085 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚328280563095040 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 161234574 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚72573330 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚72573332 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚287119666767727 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4012709898752 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚72573330 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4012709898752 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26087 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1187 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚293475643697152 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177046528 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚88661242 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 233054720 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚283121942597104 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚88661242 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26089 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410304 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 _runtime‚4097.516619773 _step‚287 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚293475643697152 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31930 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917410304 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚289462951801344 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31685 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚21111792 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 % _timestamp‚1.7885144720461981e+09 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚45130295 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚283121925557744ú{ù¨÷'öñO _runtime‚4097.516619773 _step‚287 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚293475643697152 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31930 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917410304 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚289462951801344 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31685 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚21111792 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 % _timestamp‚1.7885144720461981e+09 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚45130295 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚283121925557744 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007237 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 161208485 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚289462933798400 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4013609306112 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚66242085 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚328280563095040 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 161234574 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚72573330 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚72573332 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚287119666767727 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4012709898752 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚72573330 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4012709898752 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26087 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1187 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚293475643697152 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177046528 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚88661242 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 233054720 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚283121942597104 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚88661242 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26089 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410304 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171ŸI¹É2÷- _wandbruntime‚4098  _runtime‚4098T†”Þ·øöM [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚289542582096896 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚289542564093952 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚289542564093952 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚609 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 161279089 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007237 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4014722122752 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚45142671 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚607 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955590656 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚293557304219648 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚72593228 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚283199905651872 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411624 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚88685859 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214661 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4013822702080 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚287199669974157 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚287198755523173 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚283199922691232 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 242301440 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _runtime‚4098.493278386 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚88685859 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚328370800958976 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899420672 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚66260877 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31930 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917423616 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚21118206 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚324179509511680 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 % _timestamp‚1.7885144730221632e+09 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚66260268 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚324179485571584 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 218361344 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1187 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917423616 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 398533632 trainer/global_step‚3549 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚289542582096896 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4013822702080 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26115 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26113 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚88659746 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚293557304219648 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚293556386796032 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚45142669 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3998849871301’.›'ùûM H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 161279089 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917423616 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 398533632 trainer/global_step‚3549 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚289542582096896 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4013822702080 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26115 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26113 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚88659746 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚293557304219648 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚293556386796032 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚45142669 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3998849871301 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚289542582096896 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚289542564093952 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚289542564093952 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚609 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 161279089 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007237 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4014722122752 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚45142671 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚607 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955590656 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚293557304219648 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚72593228 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚283199905651872 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411624 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚88685859 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214661 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4013822702080 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚287199669974157 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚287198755523173 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚283199922691232 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 242301440 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _runtime‚4098.493278386 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚88685859 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚328370800958976 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899420672 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚66260877 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31930 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917423616 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚21118206 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚324179509511680 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 % _timestamp‚1.7885144730221632e+09 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚66260268 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚324179485571584 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 218361344 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1187 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 FDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177046528 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1212 % _timestamp‚1.7885145249324305e+09 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚89868057 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚287020930264144 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007237 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚297512813054976 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚332804705534464 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 163436287 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚73568230 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26089 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚89894144 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31685 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214661 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31930 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚67150097 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚332804454831104 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917410304 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410304 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚89868057 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚291074118131235 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31930 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚297513730465280 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚293444466578944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177046528 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4248327443456 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _runtime‚4150.402870759 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚73568232 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚297513730465280 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚297512813054976 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411416 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955590656 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1187 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚293444448576000 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 226763264 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26087 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4069263886336 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31685 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚328556151327744 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚636 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450776 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚89894144 SED¿] ,PeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 398533632Js÷'ˆñO PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚73568230 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚67150733 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 trainer/global_step‚3599 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4068364478976 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407360 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚45749093 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚21401638 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚297512813054976 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚332804705534464 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 163436287 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚73568230 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26089 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚89894144 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31685 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214661 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31930 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚67150097 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚332804454831104 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917410304 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410304 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚89868057 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚291074118131235 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31930 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚297513730465280 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚293444466578944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177046528 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4248327443456 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _runtime‚4150.402870759 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚73568232 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚297513730465280 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚297512813054976 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411416 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955590656 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1187 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚293444448576000 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 226763264 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26087 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4069263886336 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31685 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚328556151327744 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚636 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450776 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚89894144 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 398533632 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26087 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚45749095 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 250703360 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177046528 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1212 % _timestamp‚1.7885145249324305e+09 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚89868057 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚287020930264144 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007237 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 JDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 163506891 $ _timestamp‚1.788514526003119e+09 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚297594473577472 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899420672 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4069477282304 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4069477282304 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚73588130 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚73588128 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 163506891 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚89892648 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚297594473577472 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚293524096874496 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 163480776 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177046528 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚328645446608384 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚67168166 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4249489343488 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚293524096874496 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31930 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚293524078871552 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 226749952 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚291154121337665 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚293524078871552 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955590656 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 163480776 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4054296528409 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177046528 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚45761469 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚332895186641920 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214661 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚89918761 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26113 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚73588130 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955590656 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917423616 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚89892648 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4055193940033 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917423616 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚621 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 250690048 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚291153206886681 _step‚292 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899420672 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚287098910358272 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚89918761 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚332894935951872 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26113 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31685 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚67168789 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1187 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4249716093440 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚21406697 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚623 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007237 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31930HkæÙ'‹úM BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚67168166 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4249489343488 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚293524096874496 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31930 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚293524078871552 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 226749952 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚291154121337665 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚293524078871552 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955590656 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 163480776 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4054296528409 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177046528 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚45761469 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚332895186641920 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214661 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚89918761 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26113 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚73588130 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955590656 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917423616 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚89892648 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4055193940033 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917423616 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚621 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 250690048 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚291153206886681 _step‚292 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899420672 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚287098910358272 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚89918761 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚332894935951872 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26113 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31685 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚67168789 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1187 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4249716093440 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚21406697 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚623 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007237 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31930 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 163506891 $ _timestamp‚1.788514526003119e+09 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚297594473577472 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899420672 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4069477282304 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4069477282304 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚73588130 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚73588128 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 163506891 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚89892648 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚297594473577472 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚293524096874496 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 163480776 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177046528 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚328645446608384 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944¤hfŸn2Œ- _wandbruntime‚4151  _runtime‚4151#+L\Ö 1loss/SRqqt1/detection_loss‚1.9735387576123078 2loss/SRqqt3/assignment_loss‚1.3100263957372482 1loss/FBt2/assignment_loss‚0.21138926910156497 .loss/FRt1/detection_loss‚1.979708292449208 .loss/SRqqt2/detection_loss‚1.0197716136692 /loss/FBt3/detection_loss‚2.4198607206344604 trainer/global_step‚3599 0loss/FRt2/assignment_loss‚1.1945951203910672 2loss/SRqqt4/assignment_loss‚1.9574814140796661 1loss/SRqqt4/detection_loss‚2.5386481073594864 2loss/SRqqt2/assignment_loss‚0.9836325450021712 1loss/FBt1/assignment_loss‚0.24013267686111955 1loss/FBt3/assignment_loss‚0.15519543527511814 0loss/FBt4/assignment_loss‚0.2666338298909573 .loss/FBt1/detection_loss‚2.027481277783712 /loss/FBt4/detection_loss‚2.7559520271089344 0loss/FRt3/assignment_loss‚2.7898901891708374 %loss/total_loss‚39.74948789287581 _runtime‚4151.475481778 0loss/FRt4/assignment_loss‚2.6445610910654067 2loss/SRqqt1/assignment_loss‚1.2847381779214455 epoch‚2 % _timestamp‚1.7885145260051658e+09 _step‚293 /loss/FRt2/detection_loss‚1.5476355450798054 .loss/FRt3/detection_loss‚2.101130429506302 1loss/SRqqt3/detection_loss‚1.6193340663962505 /loss/FBt2/detection_loss‚0.5695246960260446 0loss/FRt1/assignment_loss‚2.7356045137752187 /loss/FRt4/detection_loss‚3.4230217009782793ÎÉ4bÆŽÛ 0loss/FRt4/assignment_loss‚2.6445610910654067 2loss/SRqqt1/assignment_loss‚1.2847381779214455 epoch‚2 % _timestamp‚1.7885145260051658e+09 _step‚293 /loss/FRt2/detection_loss‚1.5476355450798054 .loss/FRt3/detection_loss‚2.101130429506302 1loss/SRqqt3/detection_loss‚1.6193340663962505 /loss/FBt2/detection_loss‚0.5695246960260446 0loss/FRt1/assignment_loss‚2.7356045137752187 /loss/FRt4/detection_loss‚3.4230217009782793 1loss/SRqqt1/detection_loss‚1.9735387576123078 2loss/SRqqt3/assignment_loss‚1.3100263957372482 1loss/FBt2/assignment_loss‚0.21138926910156497 .loss/FRt1/detection_loss‚1.979708292449208 .loss/SRqqt2/detection_loss‚1.0197716136692 /loss/FBt3/detection_loss‚2.4198607206344604 trainer/global_step‚3599 0loss/FRt2/assignment_loss‚1.1945951203910672 2loss/SRqqt4/assignment_loss‚1.9574814140796661 1loss/SRqqt4/detection_loss‚2.5386481073594864 2loss/SRqq –,Œ›t2/assignment_loss‚0.9836325450021712 1loss/FBt1/assignment_loss‚0.24013267686111955 1loss/FBt3/assignment_loss‚0.15519543527511814 0loss/FBt4/assignment_loss‚0.2666338298909573 .loss/FBt1/detection_loss‚2.027481277783712 /loss/FBt4/detection_loss‚2.7559520271089344 0loss/FRt3/assignment_loss‚2.7898901891708374 %loss/total_loss‚39.74948789287581 _runtime‚4151.475481778¥!PÙý’‚(:‡ á¡êÔ¦§ÿ gpu.0.gpu‚35.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚168.992' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.248 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0³fø\‚(:Ñ á¡êÔƼ€ disk.sda1.out‚0 network.sent‚ 8.318829e+06( disk./.usagePercent‚75.8295402359365# disk.nvme0n1.out‚21715.99609375 disk.sda2.out‚ 478.380859375( proc.memory.availableMB‚ 732.94140625 disk.sdb.out‚ 478.380859375 disk.nvme1n1.in‚ 321.4609375 disk.sdb1.in‚0* proc.memory.percent‚17.775559425354004 proc.cpu.threads‚17 disk.nvme0n1.in‚ 124.28125 disk.rbd0.in‚0 disk.sdb2.out‚ 478.380859375 disk.dm-0.out‚46278.97265625 disk.sda.out‚ 478.380859375 disk.rbd2.in‚0 disk.md0p1.in‚ 0.17578125 disk.sda.in‚0.09375" disk.nvme1n1.out‚ 24562.9765625 disk.md0.in‚ 0.17578125$ memory_percent‚95.52648067474365# proc.memory.rssMB‚ 2912.34765625 cpu‚25.165666776651616 disk.sdb1.out‚0 disk.dm-0.in‚ 445.7421875 network.recv‚ 1.393279e+06 disk.rbd2.out‚1.0625 disk.sdb2.in‚ 0.08203125% disk./.usageGB‚2712.0316581726074 disk.md0p1.out‚ 460.99609375 disk.sdb.in‚ 0.08203125 disk.sda2.in‚0.09375 disk.rbd0.out‚1.125 disk.md0.out‚ 460.99609375 disk.sda1.in‚0­ˆÁ”‘‚(:‰ ð¡êÔÚÎÆÿ gpu.0.gpu‚42.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚166.207' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚41.55175 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¢Ævsc’‚(:Ø ð¡êÔĬɀ network.sent‚ 8.426468e+06 disk.dm-0.in‚ 544.94921875$ memory_percent‚95.52488327026367 disk.sda.in‚0.09375 disk.sda2.in‚0.09375 disk.rbd0.out‚1.125 disk.nvme1n1.in‚ 367.9453125 network.recv‚ 1.398383e+06$ disk./.usageGB‚2712.776752471924 disk.nvme0n1.in‚ 177.00390625# disk.nvme0n1.out‚21869.10546875 disk.sdb.out‚480.42626953125 disk.sda.out‚480.42626953125! disk.sda2.out‚480.42626953125 disk.dm-0.out‚46663.52734375 disk.rbd0.in‚0 disk.sdb1.out‚0 disk.rbd2.in‚0 disk.sda1.out‚0# proc.memory.rssMB‚ 2912.34765625 cpu‚25.1651334958553! disk.nvme1n1.out‚ 24794.421875 disk.md0.out‚ 462.96484375 disk.rbd2.out‚1.0625& proc.memory.availableMB‚ 733.203125 proc.cpu.threads‚17) disk./.usagePercent‚75.85037338439159 disk.md0p1.in‚ 0.17578125 disk.sdb.in‚ 0.08203125! disk.sdb2.out‚480.42626953125 disk.sdb1.in‚0 disk.md0.in‚ 0.17578125 disk.sdb2.in‚ 0.08203125* proc.memory.percent‚17.775559425354004 disk.md0p1.out‚ 462.96484375 disk.sda1.in‚0wðUp’“‚(:‡ ÿ¡êÔÙŽ‘‚ gpu.0.gpu‚7.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚37.0 gpu.0.powerWatts‚183.258' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚45.8145 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ãö9BX”‚(:Í ÿ¡êÔ•Æׄ disk.md0.in‚ 0.17578125 disk.sdb2.in‚ 0.08203125 disk.sda1.out‚0 proc.cpu.threads‚17 disk.dm-0.in‚ 635.04296875 disk.nvme0n1.in‚ 238.8515625 disk.sda.out‚481.9580078125* proc.memory.percent‚18.256020545959473 cpu‚25.1674622503174) disk./.usagePercent‚75.85262136176658$ disk./.usageGB‚2712.857151031494 disk.md0p1.in‚ 0.17578125 disk.sdb.out‚481.9580078125 disk.sdb1.in‚0 disk.md0p1.out‚ 464.42578125 disk.sda2.out‚481.9580078125 disk.nvme1n1.in‚ 396.19140625 disk.rbd2.out‚1.0625 disk.sdb2.out‚481.9580078125 disk.sda.in‚0.09375 network.sent‚ 8.434514e+06 disk.nvme0n1.out‚ 21893.96875 disk.rbd0.out‚1.125 disk.sda1.in‚0 disk.sda2.in‚0.09375 disk.rbd0.in‚0 disk.dm-0.out‚ 47061.4375 disk.md0.out‚ 464.42578125 disk.rbd2.in‚0$ memory_percent‚96.00863456726074# proc.memory.rssMB‚ 2991.06640625 disk.sdb.in‚ 0.08203125 disk.nvme1n1.out‚ 25167.46875' proc.memory.availableMB‚ 653.9453125 network.recv‚ 1.402166e+06 disk.sdb1.out‚0SÄìÖ”•‚(:‰ Ž¢êÔÚ¦þ gpu.0.gpu‚55.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚181.699' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚45.42475 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0xP'R–‚(:Ç Ž¢êÔ­ê× disk.sdb.in‚ 0.08203125 disk.md0.out‚ 465.90625 disk.rbd0.in‚0 disk.rbd0.out‚1.125 disk.sdb1.out‚0 proc.cpu.threads‚17) disk./.usagePercent‚75.84848506633091 disk.md0p1.out‚ 465.90625* proc.memory.percent‚18.256020545959473 network.sent‚ 8.438732e+06 disk.sdb1.in‚0 disk.md0p1.in‚ 0.17578125# disk.nvme0n1.out‚22299.12109375 disk.sda.out‚ 483.51171875 disk.sdb2.in‚ 0.08203125 disk.sdb2.out‚ 483.51171875 disk.sda1.in‚0 disk.nvme0n1.in‚ 238.8515625# memory_percent‚96.0078477859497# proc.memory.rssMB‚ 2991.06640625$ disk./.usageGB‚2712.709217071533 disk.nvme1n1.out‚ 25398.03125 disk.md0.in‚ 0.17578125 disk.dm-0.in‚ 635.04296875 cpu‚25.18063179855866 network.recv‚ 1.406867e+06 disk.sda2.out‚ 483.51171875 disk.nvme1n1.in‚ 396.19140625 disk.sdb.out‚ 483.51171875 disk.rbd2.out‚1.0625 disk.sda.in‚0.09375 disk.sda2.in‚0.09375 disk.rbd2.in‚0 disk.sda1.out‚0 disk.dm-0.out‚47697.15234375( proc.memory.availableMB‚ 654.07421875E›¤n”—‚(:‰ ¢êÔŸ¶›þ gpu.0.gpu‚59.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚182.641' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚45.66025 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚01¶²_˜‚(:Ô ¢êÔê–Ù! disk.sdb2.out‚485.64306640625! disk.sda2.out‚485.64306640625 disk.nvme1n1.in‚ 403.75390625) disk./.usagePercent‚75.83382372290653$ disk./.usageGB‚2712.184856414795# disk.nvme0n1.out‚22357.47265625 disk.sda2.in‚0.09375) proc.memory.percent‚18.25864315032959 disk.rbd2.in‚0 proc.cpu.threads‚17 disk.md0p1.out‚ 467.984375 disk.rbd0.out‚1.125# disk.nvme1n1.out‚25603.24609375 disk.md0.in‚ 0.17578125 disk.md0.out‚ 467.984375 disk.rbd2.out‚1.0625 disk.dm-0.out‚ 47960.71875 disk.nvme0n1.in‚ 316.828125 disk.sdb1.out‚0 network.sent‚ 8.44397e+06 disk.dm-0.in‚ 720.58203125 disk.md0p1.in‚ 0.17578125 disk.sdb.out‚485.64306640625 disk.sda.out‚485.64306640625( proc.memory.availableMB‚ 653.91015625 disk.sda1.out‚0 disk.sda.in‚0.09375 disk.rbd0.in‚0 disk.sdb1.in‚0$ memory_percent‚96.00884914398193# proc.memory.rssMB‚ 2991.49609375 network.recv‚ 1.410177e+06 disk.sdb2.in‚ 0.08203125 disk.sda1.in‚0 disk.sdb.in‚ 0.08203125 cpu‚25.193022253457748¢6ln“™‚(:ˆ ¬¢êÔ¡§Ÿÿ gpu.0.gpu‚31.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚36.0 gpu.0.powerWatts‚113.798' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.4495 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0†/IWš‚(:Ì ¬¢êԈᮀ disk.sdb.in‚ 0.08203125 network.sent‚ 8.451153e+06 disk.dm-0.in‚ 883.40234375 disk.dm-0.out‚48074.34765625 disk.md0p1.in‚ 0.17578125 disk.sda.out‚ 487.232421875 disk.sdb1.out‚0 disk.sda1.out‚0 disk.rbd2.in‚0 disk.md0p1.out‚ 469.5078125! disk.nvme1n1.out‚ 25715.578125 disk.md0.out‚ 469.5078125 disk.sdb2.out‚ 487.232421875 disk.sda.in‚0.09375 disk.rbd0.in‚0 disk.rbd0.out‚1.125 disk.nvme1n1.in‚497.375 disk.md0.in‚ 0.17578125( proc.memory.availableMB‚ 654.05078125 network.recv‚ 1.413661e+06 disk.sda2.in‚0.09375$ memory_percent‚96.00799083709717% disk./.usageGB‚2712.1404991149902 disk.sda1.in‚0 disk.nvme0n1.in‚ 386.02734375# disk.nvme0n1.out‚22358.76953125 disk.sda2.out‚ 487.232421875 disk.sdb1.in‚0 disk.rbd2.out‚1.0625' disk./.usagePercent‚75.832583474239 disk.sdb.out‚ 487.232421875# proc.memory.rssMB‚ 2991.49609375) proc.memory.percent‚18.25864315032959 cpu‚25.205459216100014 proc.cpu.threads‚17 disk.sdb2.in‚ 0.08203125øÑÕC2›- _wandbruntime‚4240  _runtime‚4240а+‘Y*œÓT bNDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.allocated‚298299107367424 YLDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.current‚18002944 ZLDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.current‚ 675116032 hVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.allocated‚ 4305310224384 LFDeviceStatsMonitor.on_validation_batch_start/oversize_segments.current‚0 TNDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.current‚2 MCDeviceStatsMonitor.on_validation_batch_start/allocation.all.current‚20520 MDDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.peak‚1212 \JDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.freed‚ 4123706631680 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.freed‚46558181 ICDeviceStatsMonitor.on_validation_batch_start/oversize_segments.peak‚0 eQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.allocated‚291643124340464 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 [LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.peak‚ 1177046528 UEDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.peak‚ 13955590656 J@DeviceStatsMonitor.on_validation_batch_start/allocation.all.peak‚31930 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.allocated‚46558183 XJDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.peak‚ 398533632 `PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 ^NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.current‚ 13037993984 KEDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.freed‚302422795996160 YLDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.allocated‚91013091 jVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.allocated‚333774113636352 E?DeviceStatsMonitor.on_validation_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_validation_batch_start/num_sync_all_streams‚2 UEDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.peak‚ 13510214661 [LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.peak‚ 1175007237 YIDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.peak‚ 12835443712 JDDeviceStatsMonitor.on_validation_batch_start/oversize_segments.freed‚0 YIDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.allocated‚ 14220787712 [KDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.freed‚0 I@DeviceStatsMonitor.on_validation_batch_start/segment.all.current‚1171 _step‚294 SEDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.current‚ 693118976 QDDeviceStatsMonitor.on_validation_batch_start/active.small_pool.freed‚90992573 VHDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.current‚ 693118976 UHDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.freed‚74873898 \LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.peak‚ 12392107008 I?DeviceStatsMonitor.on_validation_batch_start/active.all.current‚20520 OGDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.current‚204 fRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.freed‚333774089696256 K=DeviceStatsMonitor.on_validation_batch_start/active.all.freed‚ 165866471 cQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.allocated‚ 4124381747712 UHDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.freed‚90992573 cODeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.allocated‚338079423860736 _MDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.freed‚ 4108334963327 TDDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.peak‚ 14220787712 ZFDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.freed‚302422795996160 PGDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.peak‚3677 TJDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.current‚20518 PFDeviceStatsMonitor.on_validation_batch_start/active.small_pool.current‚20518 F=DeviceStatsMonitor.on_validation_batch_start/num_device_alloc‚1171 QIDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.allocated‚607 OADeviceStatsMonitor.on_validation_batch_start/active.all.allocated‚ 165886991 VNDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.current‚202 [MDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.current‚ 173004800 bTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.current‚ 149064704 UHDeviceStatsMonitor.on_validation_batch_start/active.small_pool.allocated‚91013091 dRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.freed‚ 4305161159680 ]ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.current‚ 673144756 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.freed‚0 XIDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.peak‚ 1177046528 ]ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.current‚ 675116032 LFDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.peak‚0 % _timestamp‚1.7885146147564163e+09 [GDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.allocated‚302423489115136 ^JDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.allocated‚302423489115136 REDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.freed‚68385689 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 B;DeviceStatsMonitor.on_validation_batch_start/max_split_size‚-1 ;5DeviceStatsMonitor.on_validation_batch_start/num_ooms‚0 QGDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.peak‚31685 F=DeviceStatsMonitor.on_validation_batch_start/segment.all.peak‚1171 \LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 _MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.freed‚ 4123706631680 RKDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.peak‚35 _KDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.freed‚338079250855936 aTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.current‚23940096 OIDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 LFDeviceStatsMonitor.on_validation_batch_start/active.large_pool.current‚2 TKDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.peak‚1187 LDDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.peak‚607 QDDeviceStatsMonitor.on_validation_batch_start/active.large_pool.freed‚74873898 KBDeviceStatsMonitor.on_validation_batch_start/segment.all.allocated‚1171 ZKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 PJDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.current‚2 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.allocated‚21827710 \ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.current‚17039360 ]NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.current‚ 1182793728 LDDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.peak‚564 trainer/global_step‚3626 LCDeviceStatsMonitor.on_validation_batch_start/active.large_pool.peak‚3677 ^JDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.freed‚298299089364480 YLDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.allocated‚74873900 D>DeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 ZFDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.freed‚295751442264431 _runtime‚4240.227621355 `NDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.allocated‚ 4124381747712 SEDeviceStatsMonitor.on_validation_batch_start/allocation.all.allocated‚ 165886991 _PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 QIDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.allocated‚564 OGDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.current‚564 KEDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.freed‚0 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.freed‚21827508 NHDeviceStatsMonitor.on_validation_batch_start/oversize_segments.allocated‚0 cQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.allocated‚ 4109008108083 OGDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.current‚607 RBDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.peak‚ 13955590656 \ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.allocated‚68385893 QKDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.allocated‚0 aMDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.freed‚298299089364480 aMDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.freed‚291643107301104 UHDeviceStatsMonitor.on_validation_batch_start/active.large_pool.allocated‚74873900 OADeviceStatsMonitor.on_validation_batch_start/allocation.all.freed‚ 165866471 VHDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.current‚ 690184116 MCDeviceStatsMonitor.on_validation_batch_start/active.small_pool.peak‚31685 BDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 LFDeviceStatsMonitor.on_validation_batch_start/active.large_pool.current‚2 TKDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.peak‚1187 LDDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.peak‚607 QDDeviceStatsMonitor.on_validation_batch_start/active.large_pool.freed‚74873898 KBDeviceStatsMonitor.on_validation_batch_start/segment.all.allocated‚1171 ZKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 PJDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.current‚2 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.allocated‚21827710 \ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.current‚17039360 ]NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.current‚ 1182793728 LDDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.peak‚564 trainer/global_step‚3626 LCDeviceStatsMonitor.on_validation_batch_start/active.large_pool.peak‚3677 ^JDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.freed‚298299089364480 YLDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.allocated‚74873900 D>DeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 ZFDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.freed‚295751442264431 _runtime‚4240.227621355 `NDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.allocated‚ 4124381747712 SEDeviceStatsMonitor.on_validation_batch_start/allocation.all.allocated‚ 165886991 _PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 QIDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.allocated‚564 OGDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.current‚564 KEDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.freed‚0 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.freed‚21827508 NHDeviceStatsMonitor.on_validation_batch_start/oversize_segments.allocated‚0 cQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.allocated‚ 4109008108083 OGDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.current‚607 RBDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.peak‚ 13955590656 \ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.allocated‚68385893 QKDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.allocated‚0 aMDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.freed‚298299089364480 aMDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.freed‚291643107301104 UHDeviceStatsMonitor.on_validation_batch_start/active.large_pool.allocated‚74873900 OADeviceStatsMonitor.on_validation_batch_start/allocation.all.freed‚ 165866471 VHDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.current‚ 690184116 MCDeviceStatsMonitor.on_validation_batch_start/active.small_pool.peak‚31685 BDeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚31930 WJDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.allocated‚74889176 OEDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.peak‚31685 bPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.freed‚ 4305649722368 \HDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.freed‚298352660640256 WJDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.allocated‚91021865 YKDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.current‚ 173004800 D;DeviceStatsMonitor.on_validation_batch_end/num_device_alloc‚1171 XIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 cODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.allocated‚298352678643200 GADeviceStatsMonitor.on_validation_batch_end/oversize_segments.peak‚0 NEDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.peak‚3677 dPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.freed‚333829539168256 _step‚295 aMDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.allocated‚338135361895424 `RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.current‚ 149064704 ZJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.peak‚ 12392107008 G>DeviceStatsMonitor.on_validation_batch_end/segment.all.current‚1171 YEDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.allocated‚302477548953600 NHDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.current‚2 @9DeviceStatsMonitor.on_validation_batch_end/max_split_size‚-1 OGDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.allocated‚564 [LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.current‚ 1182793728 RLDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.current‚2 SFDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.freed‚74889174 SFDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.freed‚91001347 RIDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.peak‚1187 E?DeviceStatsMonitor.on_validation_batch_end/num_sync_all_streams‚2 ICDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.freed‚0 ZJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 OIDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.allocated‚0 YIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 NDDeviceStatsMonitor.on_validation_batch_end/active.small_pool.current‚20518 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.allocated‚21833208 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 LFDeviceStatsMonitor.on_validation_batch_end/oversize_segments.allocated‚0 _KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.freed‚298352660640256 aODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.allocated‚ 4124870310400 cODeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.allocated‚291692894430288 ICDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.freed‚0 ZMDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.current‚18002944 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.freed‚21833006 SCDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.peak‚ 13510214661 ZMDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.current‚ 14220787712 OBDeviceStatsMonitor.on_validation_batch_end/active.small_pool.freed‚91001347 \HDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.allocated‚302477548953600 _KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.freed‚291692877390928 ]KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.freed‚ 4108821890051 \LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.current‚ 13037993984 JDDeviceStatsMonitor.on_validation_batch_end/active.large_pool.current‚2 TFDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.current‚ 693118976 MEDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.current‚607 SCDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.peak‚ 13955590656 KADeviceStatsMonitor.on_validation_batch_end/active.small_pool.peak‚31685 UADeviceStatsMonitor.on_validation_batch_end/active_bytes.all.freed‚302476855834624 [MDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.current‚ 675116032 TLDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.current‚202 JDDeviceStatsMonitor.on_validation_batch_end/oversize_segments.current‚0 I@DeviceStatsMonitor.on_validation_batch_end/segment.all.allocated‚1171 SFDeviceStatsMonitor.on_validation_batch_end/active.large_pool.allocated‚74889176 JADeviceStatsMonitor.on_validation_batch_end/active.large_pool.peak‚3677 XDDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.freed‚302476855834624 fTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.allocated‚ 4305798787072 TFDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.current‚ 690184116 MEDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.current‚564 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.freed‚46567621 PIDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.peak‚35 ZHDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.freed‚ 4124195194368 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.freed‚0 ]NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 P@DeviceStatsMonitor.on_validation_batch_end/active_bytes.all.peak‚ 13955590656 RHDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.current‚20518 hTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.allocated‚333829563108352 KEDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.freed‚0 XDDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.freed‚295801699280979 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.freed‚0 WGDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.peak‚ 12835443712 QCDeviceStatsMonitor.on_validation_batch_end/allocation.all.allocated‚ 165911041 VHDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.peak‚ 398533632 93DeviceStatsMonitor.on_validation_batch_end/num_ooms‚0 WGDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.allocated‚ 14220787712 JBDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.peak‚564 M?DeviceStatsMonitor.on_validation_batch_end/allocation.all.freed‚ 165890521 TGDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.allocated‚68400831 JBDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.peak‚607 MEDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.current‚204 @:DeviceStatsMonitor.on_validation_batch_end/num_device_free‚0 trainer/global_step‚3626 RBDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.peak‚ 14220787712 KBDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.peak‚1212 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.allocated‚46567623 `LDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.allocated‚298352678643200 C=DeviceStatsMonitor.on_validation_batch_end/num_oom_rejections‚0 [MDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.current‚ 673144756 BDeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚31930 WJDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.allocated‚74889176 OEDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.peak‚31685 bPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.freed‚ 4305649722368 \HDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.freed‚298352660640256 WJDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.allocated‚91021865 YKDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.current‚ 173004800 D;DeviceStatsMonitor.on_validation_batch_end/num_device_alloc‚1171 XIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 cODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.allocated‚298352678643200 GADeviceStatsMonitor.on_validation_batch_end/oversize_segments.peak‚0 NEDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.peak‚3677 dPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.freed‚333829539168256 _step‚295 aMDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.allocated‚338135361895424 `RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.current‚ 149064704 ZJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.peak‚ 12392107008 G>DeviceStatsMonitor.on_validation_batch_end/segment.all.current‚1171 YEDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.allocated‚302477548953600 NHDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.current‚2 @9DeviceStatsMonitor.on_validation_batch_end/max_split_size‚-1 OGDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.allocated‚564 [LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.current‚ 1182793728 RLDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.current‚2 SFDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.freed‚74889174 SFDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.freed‚91001347 RIDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.peak‚1187 E?DeviceStatsMonitor.on_validation_batch_end/num_sync_all_streams‚2 ICDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.freed‚0 ZJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 OIDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.allocated‚0 YIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 NDDeviceStatsMonitor.on_validation_batch_end/active.small_pool.current‚20518 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.allocated‚21833208 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 LFDeviceStatsMonitor.on_validation_batch_end/oversize_segments.allocated‚0 _KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.freed‚298352660640256 aODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.allocated‚ 4124870310400 cODeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.allocated‚291692894430288 ICDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.freed‚0 ZMDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.current‚18002944 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.freed‚21833006 SCDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.peak‚ 13510214661 ZMDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.current‚ 14220787712 OBDeviceStatsMonitor.on_validation_batch_end/active.small_pool.freed‚91001347 \HDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.allocated‚302477548953600 _KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.freed‚291692877390928 ]KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.freed‚ 4108821890051 \LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.current‚ 13037993984 JDDeviceStatsMonitor.on_validation_batch_end/active.large_pool.current‚2 TFDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.current‚ 693118976 MEDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.current‚607§MdŸa¡‚(:Ö »¢êÔ£ºÎ disk.rbd0.out‚1.125 disk.sdb1.out‚0 disk.md0.in‚ 0.17578125 disk.rbd2.in‚0$ memory_percent‚96.00934982299805 disk.md0p1.out‚ 471.05859375 disk.sdb1.in‚0 disk.rbd2.out‚1.0625 disk.dm-0.out‚48366.98828125) proc.memory.percent‚18.25864315032959 cpu‚25.219006528933484 proc.cpu.threads‚17 disk.md0p1.in‚ 0.17578125 disk.sdb.in‚ 0.08203125 disk.sda.out‚488.85693359375! disk.sdb2.out‚488.85693359375 disk.nvme1n1.in‚ 498.61328125 disk.md0.out‚ 471.05859375# proc.memory.rssMB‚ 2991.49609375 network.sent‚ 8.454928e+06 disk.sdb.out‚488.85693359375 disk.sda1.in‚0 network.recv‚ 1.418233e+06 disk.nvme0n1.out‚ 22363.96875 disk.sda.in‚0.09375 disk.sda2.in‚0.09375 disk.rbd0.in‚0# disk.nvme1n1.out‚26003.01953125 disk.sdb2.in‚ 0.08203125( disk./.usagePercent‚75.8381396346783$ disk./.usageGB‚2712.339214324951 disk.nvme0n1.in‚ 393.265625 disk.sda1.out‚0 disk.dm-0.in‚ 891.87890625& proc.memory.availableMB‚ 653.828125! disk.sda2.out‚488.85693359375™OQc“¢‚(:ˆ »¢êÔƦÀ gpu.0.gpu‚1.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚35.0 gpu.0.powerWatts‚113.181' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.29525 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ï_bi2£- _wandbruntime‚4257  _runtime‚4257Hæ—Š,¤ò¥ A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 ("Purity/*FRt3SRqqt0FBt/event_purity‚0 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9905070662498474 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 % _timestamp‚1.7885146318226335e+09 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.11347517371177673 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9812901020050049 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 & Purity/4FRt1SRqqt0FBt/FRt_purity‚0 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 *jet/accuracy_1_of_4‚0.9138723611831665 jet/accuracy_5_of_5‚0 particle/accuracy_4_of_4‚1 :"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.43246185779571533 *"Purity/1FRt1SRqqt3FBt/SRqqt_purity‚0.5 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 jet/accuracy_3_of_7‚0 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.3913237750530243 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5120976567268372 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 9"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.3799999952316284 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6939917802810669 9"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.4214477837085724 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9886109232902527 particle/accuracy_2_of_5‚1 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9977843761444092 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.5863094925880432 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.30302202701568604 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 :"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.44514375925064087 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.5102425813674927 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 particle/accuracy_2_of_3‚1 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 particle/accuracy_3_of_3‚1 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 8"Purity/*FRt0SRqqt3FBt/event_purity‚0.138888880610466 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.4643764793872833 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.2780088484287262 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.4555286467075348 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.19032923877239227 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 *jet/accuracy_1_of_3‚0.8763232231140137 particle/accuracy_1_of_1‚1 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 ."Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.46875 :"Purity/1FRt*SRqqt3FBt/event_purity‚0.02777777798473835 7 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.2380952537059784 & Purity/1FRt0SRqqt4FBt/FRt_purity‚0 +"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.75 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9219577312469482 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 particle/accuracy_1_of_5‚1 /"Purity/0FRt0SRqqt3FBt/event_purity‚0.171875 9"Purity/*FRt*SRqqt4FBt/event_purity‚0.3333333432674408 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.4926883280277252 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.3541666567325592 +"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.25 particle/accuracy_3_of_6‚1 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 +"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.75 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.21925285458564758 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.012320715934038162 ;"Purity/2FRt0SRqqt2FBt/event_purity‚0.032894738018512726 7"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.37042036652565 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 *jet/accuracy_1_of_2‚0.7602043747901917 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.20000001788139343 :"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.47545838356018066 9"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.4229365289211273 particle/accuracy_1_of_6‚1 7 Purity/0FRt2SRqqt2FBt/FBt_purity‚0.9821428656578064 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.4833333194255829 7 Purity/3FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 particle/accuracy_1_of_7‚1 8 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.47058823704719543 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.4620782434940338 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 8 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.32685187458992004 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.19421984255313873 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 & Purity/4FRt*SRqqt0FBt/FRt_purity‚0 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.25643160939216614 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.4873563051223755 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.4027777910232544 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5145612359046936 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 particle/accuracy_5_of_5‚1 particle/accuracy_4_of_6‚1 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 particle/accuracy_3_of_7‚1 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5384934544563293 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.5445294976234436 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 ("Purity/0FRt3SRqqt0FBt/event_purity‚0 ,"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.625 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.08762026578187943 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9967948794364929 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.030966700986027718 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.5370664596557617 ."Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.46875 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9954620599746704 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 6 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.251851886510849 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9778911471366882 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 9"Purity/*FRt0SRqqt4FBt/event_purity‚0.4000000059604645 :"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.48929619789123535 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9964266419410706 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9878474473953247 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 ."Purity/1FRt0SRqqt3FBt/event_purity‚0.03125 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9770114421844482 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.2225503772497177 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.13128267228603363 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.09805725514888763 6 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.259202778339386 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 +jet/accuracy_3_of_5‚0.10780210047960281 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.4181753695011139 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.30000001192092896 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.5084865689277649 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 ("Purity/*FRt3SRqqt*FBt/event_purity‚0 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5934704542160034 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.23257730901241302 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9964302182197571 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.5123955011367798 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 * Purity/*FRt*SRqqt4FBt/FBt_purity‚0.875 ("Purity/4FRt*SRqqt*FBt/SRqqt_purity‚1 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.4044667184352875 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.5714285969734192 jet/accuracy_5_of_7‚0 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9867169260978699 8 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.48903509974479675 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 *jet/accuracy_2_of_3‚0.2531278133392334 & Purity/0FRt3SRqqt1FBt/FBt_purity‚1 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.5899947285652161 :"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.41364529728889465 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8888888955116272 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.5555555820465088 ;"Purity/1FRt2SRqqt1FBt/event_purity‚0.017241379246115685 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9969576597213745 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.42Ç–DÚf044216275215149 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 9"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.3888888955116272 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 (particle/accuracy‚0.8516243100166321 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 9"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.3765992820262909 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.7222222089767456 ) Purity/*FRt1SRqqt3FBt/FRt_purity‚0.75 7 Purity/0FRt2SRqqt1FBt/FBt_purity‚0.9811320900917053 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9444444179534912 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 & Purity/*FRt3SRqqt*FBt/FBt_purity‚1 jet/accuracy_6_of_7‚0 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 :"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.46419063210487366 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 particle/accuracy_6_of_7‚1 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.7142857313156128 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 & Purity/4FRt1SRqqt*FBt/FRt_purity‚0 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.2132996767759323 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.4870940148830414 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 6 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.272972971200943 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 ) Purity/1FRt1SRqqt3FBt/FRt_purity‚0.75 & Purity/1FRt3SRqqt2FBt/FRt_purity‚0 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 :"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.47473835945129395 particle/accuracy_3_of_5‚1 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.35809561610221863 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 jet/accuracy_4_of_6‚0 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 :"Purity/0FRt2SRqqt1FBt/event_purity‚0.04716981202363968 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 particle/accuracy_4_of_5‚1 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 9"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.4699999988079071 7 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.8571428656578064 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.9166666865348816 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9838693141937256 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.4371996521949768 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.03577055409550667 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 *"Purity/0FRt0SRqqt4FBt/event_purity‚0.5 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9996392130851746 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.22920842468738556 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 *jet/accuracy_1_of_5‚0.9622887372970581 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.2121880203485489 7 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.1962696760892868 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 :"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.30000001192092896 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.3101955056190491 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9866711497306824 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.0523875392973423 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.26781609654426575 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.32053571939468384 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 jet/accuracy_1_of_6‚1 *"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.5 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.2777777910232544 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.21177342534065247 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 *particle/specificity‚0.938374936580658 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 8"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.562059223651886 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9785522222518921 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.3354561924934387 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.3638841211795807 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.22840358316898346 & Purity/1FRt2SRqqt2FBt/FBt_purity‚1 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.4692299962043762 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9933333396911621 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9125296473503113 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.17538107931613922 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.1527777761220932 ,"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.375 ("Purity/0FRt2SRqqt3FBt/SRqqt_purity‚1 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 trainer/global_step‚3626 :"Purity/0FRt1SRqqt0FBt/event_purity‚0.48145678639411926 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9734195470809937 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9925774335861206 6 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.247023805975914 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 ,jet/accuracy_4_of_4‚0.007355209439992905 particle/accuracy_2_of_4‚1 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 :"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.49438488483428955 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9774305820465088 6 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.450581431388855 8"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.675438642501831 & Purity/1FRt4SRqqt*FBt/FRt_purity‚0 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.4166666567325592 epoch‚2 :"Purity/*FRt*SRqqt1FBt/event_purity‚0.42285656929016113 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 particle/accuracy_1_of_3‚1 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.5526315569877625 7 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.2999960780143738 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.4749728739261627 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.20621643960475922 6 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.286920964717865 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.5001526474952698 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6585432887077332 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.19409538805484772 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.30272674560546875 ,jet/accuracy_4_of_5‚0.021793393418192863 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 jet/accuracy_5_of_6‚0 8"Purity/0FRt1SRqqt2FBt/event_purity‚0.061371099203825 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.44694167375564575 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.19258059561252594 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5257679224014282 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8190323710441589 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 +jet/accuracy_3_of_4‚0.06983014196157455 :"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.48145678639411926 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.21887090802192688 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.4663449227809906 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.020786091685295105 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.34652477502822876 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 7validation_average_jet_accuracy‚0.44131219387054443 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.25413885712623596 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.11270813643932343 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 *loss/val_total_loss‚42.232047115987726 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 +"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.25 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.2083333432674408 & Purity/*FRt3SRqqt2FBt/FRt_purity‚0 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2995851933956146 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.21420994400978088 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.5306524038314819 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.2212342768907547 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.30000001192092896 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 8 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.40833336114883423 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 particle/accuracy_2_of_7‚1 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.39340272545814514 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 ,"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.625 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚1 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 _step‚296 :"Purity/*FRt1SRqqt2FBt/event_purity‚0.03511904925107956 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5203084349632263 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 , Purity/*FRt2SRqqt2FBt/FRt_purity‚0.34375 & Purity/*FRt0SRqqt4FBt/FRt_purity‚0 _runtime‚4257.295347941 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.21712462604045868 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚1 7 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.3901515305042267 9"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.5632715821266174 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 ,"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.375 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 6 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.398324579000473 + Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9375 ;"Purity/2FRt*SRqqt2FBt/event_purity‚0.029069768264889717 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.7727272510528564 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 7 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.3333333432674408 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9899467825889587 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 :"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.30000001192092896 particle/accuracy_5_of_6‚1 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.18704068660736084 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 particle/accuracy_1_of_2‚1 *"Purity/0FRt*SRqqt4FBt/event_purity‚0.5 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.5350967645645142 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.5962302088737488 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.46875 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 particle/accuracy_3_of_4‚1 6 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.385416716337204 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.5316704511642456 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.5589079856872559 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.19696968793869019 particle/accuracy_5_of_7‚1 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 8 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.48331204056739807 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.19619882106781006 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.24614198505878448 7 Purity/3FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.009184248745441437 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 jet/accuracy_6_of_6‚0 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9899260401725769 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9981399774551392 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9777777194976807 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.05565476417541504 + Purity/1FRt3SRqqt1FBt/FRt_purity‚0.4375 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.5208333134651184 jet/accuracy_1_of_7‚1 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.09835425019264221 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.02303575910627842 7 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.2220400720834732 9"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.3888888955116272 & Purity/0FRt3SRqqt*FBt/FBt_purity‚1 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 & Purity/4FRt*SRqqt*FBt/FRt_purity‚0 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 8"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.497366726398468 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9913817644119263 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 particle/accuracy_4_of_7‚1 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 <"Purity/1FRt2SRqqt*FBt/event_purity‚0.0031901649199426174 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 * Purity/0FRt0SRqqt4FBt/FBt_purity‚0.875 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.21887090802192688 particle/accuracy_1_of_4‚1 particle/accuracy_7_of_7‚1 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.2818407416343689 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.4259334206581116 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9888820052146912 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 <"Purity/1FRt2SRqqt0FBt/event_purity‚0.0014430014416575432 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.16246150434017181 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9911787509918213 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9875040054321289 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.20785441994667053 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9906404614448547 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 <"Purity/1FRt1SRqqt2FBt/event_purity‚0.0069444444961845875 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.5475694537162781 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.025724366307258606 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 & Purity/1FRt4SRqqt0FBt/FRt_purity‚0 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.5331277251243591 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.015695583075284958 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 9"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.4166666567325592 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 :"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.48197826743125916 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 7 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.1905454695224762 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.5133786797523499 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.1527777761220932 7 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.2085193544626236 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9841339588165283 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 8 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.37105652689933777 particle/accuracy_2_of_6‚1 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 *"Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0.5 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.2083333432674408 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.08815628290176392 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.5601190328598022 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 +"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.75 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.23557201027870178 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.23250991106033325 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 +jet/accuracy_2_of_2‚0.11933383345603943 )jet/accuracy_2_of_5‚0.505702793598175 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.21760520339012146 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 +jet/accuracy_2_of_4‚0.39843207597732544 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 -"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.6875 :"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.43028321862220764 jet/accuracy_4_of_7‚0 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2611433267593384 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 ("Purity/0FRt3SRqqt*FBt/event_purity‚0 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.24253547191619873 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 7 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.2115471363067627 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.5334400534629822 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.8918128609657288 & Purity/1FRt*SRqqt4FBt/FRt_purity‚0 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.025101929903030396 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 9"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.4768187999725342 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5258843898773193 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 'particle/f_score‚0.4461708962917328 +jet/accuracy_3_of_6‚0.11538461595773697 jet/accuracy_7_of_7‚0 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9736841917037964 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.2639802098274231 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.5780895352363586 & Purity/*FRt3SRqqt1FBt/FBt_purity‚1 particle/accuracy_6_of_6‚1 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 *jet/accuracy_1_of_1‚0.4871956408023834 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5547760128974915 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 +"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.75 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.19088131189346313 7 Purity/1FRt0SRqqt1FBt/FBt_purity‚0.9990079402923584 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 +particle/sensitivity‚0.3829190135002136 ) Purity/2FRt3SRqqt1FBt/FRt_purity‚0.25 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.26098015904426575 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.26944443583488464 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.4097222089767456 +jet/accuracy_2_of_6‚0.49358975887298584 7 Purity/3FRt1SRqqt2FBt/FRt_purity‚0.1666666716337204 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 ("Purity/*FRt2SRqqt3FBt/SRqqt_purity‚1 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 +jet/accuracy_3_of_3‚0.02648015506565571 9"Purity/0FRt*SRqqt3FBt/event_purity‚0.1315789520740509 6 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.913194477558136 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 *"Purity/1FRt*SRqqt3FBt/SRqqt_purity‚0.5 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 particle/accuracy_2_of_2‚1 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.03898317739367485 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 8 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.20000000298023224 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 8"Purity/*FRt0SRqqt2FBt/event_purity‚0.254252165555954 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.7878787517547607 jet/accuracy_2_of_7‚0.5 & Purity/*FRt*SRqqt4FBt/FRt_purity‚0 :"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.41934871673583984 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.4285714328289032 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.1481654793024063 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 ("Purity/4FRt*SRqqt0FBt/SRqqt_purity‚1…]8¥÷¥ 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.35809561610221863 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 jet/accuracy_4_of_6‚0 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 :"Purity/0FRt2SRqqt1FBt/event_purity‚0.04716981202363968 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 particle/accuracy_4_of_5‚1 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 9"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.4699999988079071 7 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.8571428656578064 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.9166666865348816 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9838693141937256 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.4371996521949768 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.03577055409550667 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 *"Purity/0FRt0SRqqt4FBt/event_purity‚0.5 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9996392130851746 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.22920842468738556 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 *jet/accuracy_1_of_5‚0.9622887372970581 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.2121880203485489 7 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.1962696760892868 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 :"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.30000001192092896 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.3101955056190491 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9866711497306824 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.0523875392973423 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.26781609654426575 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.32053571939468384 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 jet/accuracy_1_of_6‚1 *"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.5 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.2777777910232544 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.21177342534065247 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 *particle/specificity‚0.938374936580658 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 8"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.562059223651886 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9785522222518921 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.3354561924934387 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.3638841211795807 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.22840358316898346 & Purity/1FRt2SRqqt2FBt/FBt_purity‚1 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.4692299962043762 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9933333396911621 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9125296473503113 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.17538107931613922 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.1527777761220932 ,"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.375 ("Purity/0FRt2SRqqt3FBt/SRqqt_purity‚1 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 trainer/global_step‚3626 :"Purity/0FRt1SRqqt0FBt/event_purity‚0.48145678639411926 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9734195470809937 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9925774335861206 6 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.247023805975914 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 ,jet/accuracy_4_of_4‚0.007355209439992905 particle/accuracy_2_of_4‚1 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 :"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.49438488483428955 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9774305820465088 6 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.450581431388855 8"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.675438642501831 & Purity/1FRt4SRqqt*FBt/FRt_purity‚0 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.4166666567325592 epoch‚2 :"Purity/*FRt*SRqqt1FBt/event_purity‚0.42285656929016113 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 particle/accuracy_1_of_3‚1 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.5526315569877625 7 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.2999960780143738 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.4749728739261627 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.20621643960475922 6 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.286920964717865 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.5001526474952698 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6585432887077332 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.19409538805484772 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.30272674560546875 ,jet/accuracy_4_of_5‚0Úµ_læy.021793393418192863 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 jet/accuracy_5_of_6‚0 8"Purity/0FRt1SRqqt2FBt/event_purity‚0.061371099203825 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.44694167375564575 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.19258059561252594 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5257679224014282 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8190323710441589 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 +jet/accuracy_3_of_4‚0.06983014196157455 :"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.48145678639411926 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.21887090802192688 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.4663449227809906 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.020786091685295105 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.34652477502822876 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 7validation_average_jet_accuracy‚0.44131219387054443 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.25413885712623596 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.11270813643932343 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 *loss/val_total_loss‚42.232047115987726 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 +"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.25 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.2083333432674408 & Purity/*FRt3SRqqt2FBt/FRt_purity‚0 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2995851933956146 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.21420994400978088 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.5306524038314819 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.2212342768907547 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.30000001192092896 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 8 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.40833336114883423 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 particle/accuracy_2_of_7‚1 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.39340272545814514 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 ,"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.625 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚1 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 _step‚296 :"Purity/*FRt1SRqqt2FBt/event_purity‚0.03511904925107956 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5203084349632263 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 , Purity/*FRt2SRqqt2FBt/FRt_purity‚0.34375 & Purity/*FRt0SRqqt4FBt/FRt_purity‚0 _runtime‚4257.295347941 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.21712462604045868 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚1 7 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.3901515305042267 9"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.5632715821266174 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 ,"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.375 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 6 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.398324579000473 + Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9375 ;"Purity/2FRt*SRqqt2FBt/event_purity‚0.029069768264889717 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.7727272510528564 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 7 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.3333333432674408 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9899467825889587 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 :"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.30000001192092896 particle/accuracy_5_of_6‚1 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.18704068660736084 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 particle/accuracy_1_of_2‚1 *"Purity/0FRt*SRqqt4FBt/event_purity‚0.5 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.5350967645645142 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.5962302088737488 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.46875 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 particle/accuracy_3_of_4‚1 6 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.385416716337204 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.5316704511642456 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.5589079856872559 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.19696968793869019 particle/accuracy_5_of_7‚1 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 8 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.48331204056739807 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.19619882106781006 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.24614198505878448 7 Purity/3FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.009184248745441437 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 jet/accuracy_6_of_6‚0 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9899260401725769 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9981399774551392 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9777777194976807 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.05565476417541504 + Purity/1FRt3SRqqt1FBt/FRt_purity‚0.4375 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.5208333134651184 jet/accuracy_1_of_7‚1 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.09835425019264221 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.02303575910627842 7 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.2220400720834732 9"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.3888888955116272 & Purity/0FRt3SRqqt*FBt/FBt_purity‚1 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 & Purity/4FRt*SRqqt*FBt/FRt_purity‚0 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 8"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.497366726398468 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9913817644119263 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 particle/accuracy_4_of_7‚1 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 <"Purity/1FRt2SRqqt*FBt/event_purity‚0.0031901649199426174 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 * Purity/0FRt0SRqqt4FBt/FBt_purity‚0.875 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.21887090802192688 particle/accuracy_1_of_4‚1 particle/accuracy_7_of_7‚1 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.2818407416343689 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.4259334206581116 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9888820052146912 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 <"Purity/1FRt2SRqqt0FBt/event_purity‚0.0014430014416575432 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.16246150434017181 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9911787509918213 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9875040054321289 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.20785441994667053 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9906404614448547 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 <"Purity/1FRt1SRqqt2FBt/event_purity‚0.0069444444961845875 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.5475694537162781 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.025724366307258606 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 & Purity/1FRt4SRqqt0FBt/FRt_purity‚0 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.5331277251243591 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.015695583075284958 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 9"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.4166666567325592 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 :"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.48197826743125916 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 7 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.1905454695224762 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.5133786797523499 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.1527777761220932 7 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.2085193544626236 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9841339588165283 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 8 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.37105652689933777 particle/accuracy_2_of_6‚1 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 *"Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0.5 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.2083333432674408 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.08815628290176392 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.5601190328598022 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 +"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.75 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.23557201027870178 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.23250991106033325 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 +jet/accuracy_2_of_2‚0.11933383345603943 )jet/accuracy_2_of_5‚0.505702793598175 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.21760520339012146 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 +jet/accuracy_2_of_4‚0.39843207597732544 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 -"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.6875 :"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.43028321862220764 jet/accuracy_4_of_7‚0 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2611433267593384 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 ("Purity/0FRt3SRqqt*FBt/event_purity‚0 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.24253547191619873 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 7 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.2115471363067627 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.5334400534629822 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.8918128609657288 & Purity/1FRt*SRqqt4FBt/FRt_purity‚0 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.025101929903030396 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 9"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.4768187999725342 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5258843898773193 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 'particle/f_score‚0.4461708962917328 +jet/accuracy_3_of_6‚0.11538461595773697 jet/accuracy_7_of_7‚0 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9736841917037964 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.2639802098274231 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.5780895352363586 & Purity/*FRt3SRqqt1FBt/FBt_purity‚1 particle/accuracy_6_of_6‚1 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 *jet/accuracy_1_of_1‚0.4871956408023834 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5547760128974915 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 +"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.75 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.19088131189346313 7 Purity/1FRt0SRqqt1FBt/FBt_purity‚0.9990079402923584 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 +particle/sensitivity‚0.3829190135002136 ) Purity/2FRt3SRqqt1FBt/FRt_purity‚0.25 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.26098015904426575 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.26944443583488464 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.4097222089767456 +jet/accuracy_2_of_6‚0.49358975887298584 7 Purity/3FRt1SRqqt2FBt/FRt_purity‚0.1666666716337204 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 ("Purity/*FRt2SRqqt3FBt/SRqqt_purity‚1 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 +jet/accuracy_3_of_3‚0.02648015506565571 9"Purity/0FRt*SRqqt3FBt/event_purity‚0.1315789520740509 6 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.913194477558136 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 *"Purity/1FRt*SRqqt3FBt/SRqqt_purity‚0.5 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 particle/accuracy_2_of_2‚1 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.03898317739367485 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 8 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.20000000298023224 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 8"Purity/*FRt0SRqqt2FBt/event_purity‚0.254252165555954 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.7878787517547607 jet/accuracy_2_of_7‚0.5 & Purity/*FRt*SRqqt4FBt/FRt_purity‚0 :"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.41934871673583984 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.4285714328289032 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.1481654793024063 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 ("Purity/4FRt*SRqqt0FBt/SRqqt_purity‚1 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 ("Purity/*FRt3SRqqt0FBt/event_purity‚0 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9905070662498474 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 % _timestamp‚1.7885146318226335e+09 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.11347517371177673 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9812901020050049 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 & Purity/4FRt1SRqqt0FBt/FRt_purity‚0 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 *jet/accuracy_1_of_4‚0.9138723611831665 jet/accuracy_5_of_5‚0 particle/accuracy_4_of_4‚1 :"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.43246185779571533 *"Purity/1FRt1SRqqt3FBt/SRqqt_purity‚0.5 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 jet/accuracy_3_of_7‚0 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.3913237750530243 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5120976567268372 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 9"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.3799999952316284 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6939917802810669 9"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.4214477837085724 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9886109232902527 particle/accuracy_2_of_5‚1 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9977843761444092 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.5863094925880432 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.30302202701568604 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 :"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.44514375925064087 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.5102425813674927 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 particle/accuracy_2_of_3‚1 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 particle/accuracy_3_of_3‚1 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 8"Purity/*FRt0SRqqt3FBt/event_purity‚0.138888880610466 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.4643764793872833 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.2780088484287262 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.4555286467075348 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.19032923877239227 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 *jet/accuracy_1_of_3‚0.8763232231140137 particle/accuracy_1_of_1‚1 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 ."Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.46875 :"Purity/1FRt*SRqqt3FBt/event_purity‚0.02777777798473835 7 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.2380952537059784 & Purity/1FRt0SRqqt4FBt/FRt_purity‚0 +"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.75 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9219577312469482 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 particle/accuracy_1_of_5‚1 /"Purity/0FRt0SRqqt3FBt/event_purity‚0.171875 9"Purity/*FRt*SRqqt4FBt/event_purity‚0.3333333432674408 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.4926883280277252 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.3541666567325592 +"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.25 particle/accuracy_3_of_6‚1 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 +"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.75 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.21925285458564758 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.012320715934038162 ;"Purity/2FRt0SRqqt2FBt/event_purity‚0.032894738018512726 7"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.37042036652565 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 *jet/accuracy_1_of_2‚0.7602043747901917 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.20000001788139343 :"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.47545838356018066 9"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.4229365289211273 particle/accuracy_1_of_6‚1 7 Purity/0FRt2SRqqt2FBt/FBt_purity‚0.9821428656578064 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.4833333194255829 7 Purity/3FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 particle/accuracy_1_of_7‚1 8 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.47058823704719543 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.4620782434940338 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 8 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.32685187458992004 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.19421984255313873 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 & Purity/4FRt*SRqqt0FBt/FRt_purity‚0 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.25643160939216614 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.4873563051223755 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.4027777910232544 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5145612359046936 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 particle/accuracy_5_of_5‚1 particle/accuracy_4_of_6‚1 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 particle/accuracy_3_of_7‚1 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5384934544563293 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.5445294976234436 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 ("Purity/0FRt3SRqqt0FBt/event_purity‚0 ,"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.625 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.08762026578187943 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9967948794364929 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.030966700986027718 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.5370664596557617 ."Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.46875 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9954620599746704 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 6 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.251851886510849 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9778911471366882 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 9"Purity/*FRt0SRqqt4FBt/event_purity‚0.4000000059604645 :"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.48929619789123535 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9964266419410706 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9878474473953247 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 ."Purity/1FRt0SRqqt3FBt/event_purity‚0.03125 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9770114421844482 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.2225503772497177 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.13128267228603363 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.09805725514888763 6 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.259202778339386 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 +jet/accuracy_3_of_5‚0.10780210047960281 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.4181753695011139 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.30000001192092896 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.5084865689277649 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 ("Purity/*FRt3SRqqt*FBt/event_purity‚0 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5934704542160034 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.23257730901241302 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9964302182197571 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.5123955011367798 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 * Purity/*FRt*SRqqt4FBt/FBt_purity‚0.875 ("Purity/4FRt*SRqqt*FBt/SRqqt_purity‚1 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.4044667184352875 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.5714285969734192 jet/accuracy_5_of_7‚0 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9867169260978699 8 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.48903509974479675 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 *jet/accuracy_2_of_3‚0.2531278133392334 & Purity/0FRt3SRqqt1FBt/FBt_purity‚1 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.5899947285652161 :"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.41364529728889465 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8888888955116272 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.5555555820465088 ;"Purity/1FRt2SRqqt1FBt/event_purity‚0.017241379246115685 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9969576597213745 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.4044216275215149 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 9"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.3888888955116272 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 (particle/accuracy‚0.8516243100166321 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 9"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.3765992820262909 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.7222222089767456 ) Purity/*FRt1SRqqt3FBt/FRt_purity‚0.75 7 Purity/0FRt2SRqqt1FBt/FBt_purity‚0.9811320900917053 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9444444179534912 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 & Purity/*FRt3SRqqt*FBt/FBt_purity‚1 jet/accuracy_6_of_7‚0 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 :"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.46419063210487366 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 particle/accuracy_6_of_7‚1 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.7142857313156128 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 & Purity/4FRt1SRqqt*FBt/FRt_purity‚0 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.2132996767759323 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.4870940148830414 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 6 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.272972971200943 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 ) Purity/1FRt1SRqqt3FBt/FRt_purity‚0.75 & Purity/1FRt3SRqqt2FBt/FRt_purity‚0 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 :"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.47473835945129395 particle/accuracy_3_of_5‚1¨ÝDÎZ¦‚(:Ï Ê¢êԥܫ€ network.sent‚ 8.562676e+06 disk.sda1.in‚0 disk.nvme1n1.in‚ 498.61328125 disk.nvme1n1.out‚26134.5 proc.cpu.threads‚17$ disk./.usageGB‚2712.357177734375 disk.sda1.out‚0 disk.dm-0.out‚48498.71484375 disk.sda.in‚0.09375$ memory_percent‚92.05679893493652) proc.memory.percent‚18.27082633972168 network.recv‚ 1.424189e+06 disk.rbd2.out‚1.0625! disk.sdb2.out‚490.87158203125 disk.md0p1.in‚ 0.17578125 cpu‚25.22992312130628) disk./.usagePercent‚75.83864189912404 disk.sdb1.out‚0 disk.md0.in‚ 0.17578125# disk.nvme0n1.out‚22364.21484375! disk.sda2.out‚490.87158203125( proc.memory.availableMB‚ 1301.4140625 disk.sdb2.in‚ 0.08203125 disk.dm-0.in‚ 891.87890625 disk.md0p1.out‚ 473.015625 disk.sdb1.in‚0 disk.rbd2.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚1.125 disk.sdb.in‚ 0.08203125 disk.sda.out‚490.87158203125 disk.sda2.in‚0.09375" proc.memory.rssMB‚ 2993.4921875 disk.md0.out‚ 473.015625 disk.nvme0n1.in‚ 393.265625 disk.sdb.out‚490.87158203125«iMf’§‚(:‡ Ê¢êÔ×Ë € gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚33.0 gpu.0.powerWatts‚86.589' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚21.64725 gpu.0.smClock‚1155 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ª„™»¨‚(:‰ Ù¢êÔ.´®‚¸î±ÿ gpu.0.gpu‚51.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚172.727' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.18175 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¸¬Nb©‚(:× Ù¢êÔÄöòÿ# memory_percent‚92.0382022857666 disk.md0.in‚ 0.17578125) disk./.usagePercent‚75.84379946915479) proc.memory.percent‚18.25251579284668 disk.rbd2.out‚1.0625 cpu‚25.22598726853148 proc.cpu.threads‚17 disk.md0p1.out‚ 474.60546875 disk.nvme0n1.in‚ 393.265625 disk.sdb.in‚ 0.08203125 disk.sda1.in‚0 disk.md0p1.in‚ 0.17578125! disk.sda2.out‚492.52099609375 disk.nvme1n1.in‚ 498.61328125" proc.memory.rssMB‚ 2990.4921875 network.sent‚ 8.684866e+06 disk.sda2.in‚0.09375 disk.rbd0.in‚0 disk.sdb1.in‚0 disk.sdb1.out‚0( proc.memory.availableMB‚ 1304.4609375% disk./.usageGB‚2712.5416374206543 disk.sda.out‚492.52099609375 disk.md0.out‚ 474.60546875# disk.nvme0n1.out‚23751.05078125 network.recv‚ 1.428452e+06 disk.sda1.out‚0 disk.dm-0.in‚ 891.87890625 disk.dm-0.out‚50086.64453125 disk.sda.in‚0.09375 disk.rbd0.out‚1.125 disk.sdb2.in‚ 0.08203125! disk.sdb2.out‚492.52099609375 disk.sdb.out‚492.52099609375 disk.nvme1n1.out‚ 26335.59375 disk.rbd2.in‚0|úb~2ª- _wandbruntime‚4285  _runtime‚4285×ûÇ«Á & lr-AdamW/pg1‚0.0007556111111338916 & lr-AdamW/pg2‚0.0007556111111338916 trainer/global_step‚3649 % _timestamp‚1.7885146601123471e+09 _runtime‚4285.582762678 _step‚297©µdÌ¬Æ & lr-AdamW/pg1‚0.0007556111111338916 & lr-AdamW/pg2‚0.0007556111111338916 trainer/global_step‚3649 % _timestamp‚1.7885146601123471e+09 _runtime‚4285.582762678 _step‚297©‡;d2­- _wandbruntime‚4285  _runtime‚4285Tâ<ò'®ìO YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 167181174 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177047040 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1212 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410816 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚304956686636544 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407872 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4155714906112 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4140225426604 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚294055467668816 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4155714906112 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚336515405160448 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955591168 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚91681744 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26090 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚304955769225728 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚91655656 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚340853292706816 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚300800971730432 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚300800953727488 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 167207264 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214665 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚300800971730432 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4139328015184 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚304956686636544 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411420 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 trainer/global_step‚3649 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚68988648 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚22026033 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚298194778644640 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 _runtime‚4285.584233394 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26088 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚304955769225728 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1187 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚22025414 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚298 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚68988027 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007241 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 398533632 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096>1ëú÷'¯ñO PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚91681744 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26088 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4337643134976 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚75525518 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407872 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4154815498240 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚621 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4140225426604 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚294055467668816 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4155714906112 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚336515405160448 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955591168 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚91681744 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26090 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚304955769225728 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚91655656 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚340853292706816 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚300800971730432 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚300800953727488 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 167207264 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214665 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚300800971730432 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4139328015184 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚304956686636544 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411420 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 trainer/global_step‚3649 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚68988648 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚22026033 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚298194778644640 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 _runtime‚4285.584233394 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26088 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚304955769225728 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1187 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚22025414 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚298 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚68988027 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007241 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 398533632 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 167181174 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177047040 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1212 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410816 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚304956686636544 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407872 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4155714906112 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 JDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899421184 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899421184 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚91706361 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚298273867400086 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚605 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4338810682368 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 248592384 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚336604724381184 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚294133447762944 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 $ _timestamp‚1.788514661083165e+09 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4156827722752 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚46974989 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚299 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955591168 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4156827722752 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26114 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1187 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4141334088130 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚300880602025984 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚46974991 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚22031617 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚75545418 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177047040 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4140436676502 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚340943535063552 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚305037429748736 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚69006003 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 224652288 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚294133430723584 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚300880584023040 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 398533632 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917424128 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 _runtime‚4286.554536374 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411628 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31686 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚603 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚75545416 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 167251779 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚305037429748736 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4339035334656 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚336604748321280 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450988 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚305036512324608 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 167251779 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚91680247 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚298274781851074 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚261167å S²úM XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚340943535063552 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WCDeviceStatsMonitor.on_ti>&train_batch_end/allocated_bytes.all.allocated‚305037429748736 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚69006003 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 224652288 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚294133430723584 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚300880584023040 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 398533632 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917424128 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 _runtime‚4286.554536374 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411628 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31686 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚603 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚75545416 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 167251779 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚305037429748736 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4339035334656 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚336604748321280 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450988 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚305036512324608 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 167251779 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚91680247 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚298274781851074 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚75545416 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚340943783655936 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899421184 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899421184 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚91706361 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚298273867400086 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚605 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4338810682368 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 248592384 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚336604724381184 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚294133447762944 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 $ _timestamp‚1.788514661083165e+09 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4156827722752 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚46974989 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚299 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955591168 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4156827722752 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26114 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1187 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4141334088130 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚300880602025984 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚46974991 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚22031617 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚75545418 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177047040 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4140436676502«|xÂþ2³- _wandbruntime‚4286  _runtime‚4286ä$YX´Ò .loss/FRt3/detection_loss‚2.281188746177873 1loss/SRqqt1/detection_loss‚1.6885303867624162 /loss/FBt3/detection_loss‚1.9776879946390789 0loss/FRt2/assignment_loss‚1.2961630085681348 1loss/SRqqt2/assignment_loss‚1.234329896956502 1loss/FBt2/assignment_loss‚0.05976569258454709 1loss/SRqqt4/detection_loss‚2.4868354863590665 0loss/FBt1/assignment_loss‚0.2315752334959273 -loss/FRt2/detection_loss‚1.53804212119034 trainer/global_step‚3649 % _timestamp‚1.7885146610857308e+09 2loss/SRqqt3/assignment_loss‚1.2505493928556857 .loss/FRt1/detection_loss‚1.907570221904669 .loss/FBt1/detection_loss‚2.725046396255493 /loss/FBt4/detection_loss‚3.0476460417111713 epoch‚3 /loss/FRt4/assignment_loss‚2.787734871560877 1loss/FBt3/assignment_loss‚0.20267518032163329 0loss/SRqqt3/detection_loss‚1.725026323095612 2loss/SRqqt1/assignment_loss‚1.2128340279326795 2loss/SRqqt4/assignment_loss‚2.1222192689224526 .loss/FRt4/detection_loss‚3.294195072217421 1loss/SRqqt2/detection_loss‚1.0529030379388602 _step‚300 0loss/FRt3/assignment_loss‚2.2567174226738684 0loss/FBt2/detection_loss‚0.47182243671070073 _runtime‚ 4286.55588727 %loss/total_loss‚40.41265842803063 0loss/FRt1/assignment_loss‚3.1517834703127545 0loss/FBt4/assignment_loss‚0.4098166968828688Ž_Äe]µ× 0loss/FBt1/assignment_loss‚0.2315752334959273 -loss/FRt2/detection_loss‚1.53804212119034 trainer/global_step‚3649 % _timestamp‚1.7885146610857308e+09 2loss/SRqqt3/assignment_loss‚1.2505493928556857 .loss/FRt1/detection_loss‚1.907570221904669 .loss/FBt1/detection_loss‚2.725046396255493 /loss/FBt4/detection_loss‚3.0476460417111713 epoch‚3 /loss/FRt4/assignment_loss‚2.787734871560877 1loss/FBt3/assignment_loss‚0.20267518032163329 0loss/SRqqt3/detection_loss‚1.725026323095612 2loss/SRqqt1/assignment_loss‚1.2128340279326795 2loss/SRqqt4/assignment_loss‚2.1222192689224526 .loss/FRt4/detection_loss‚3.294195072217421 1loss/SRqqt2/detection_loss‚1.0529030379388602 _step‚300 0loss/FRt3/assignment_loss‚2.2567174226738684 0loss/FBt2/detection_loss‚0.47182243671070073 _runtime‚ 4286.55588727 %loss/total_loss‚40.41265842803063 0loss/FRt1/assignment_loss‚3.1517834703127545 0loss/FBt4/assignment_loss‚0.4098166968828688 .loss/FRt3/detection_loss‚2.281188746177873 1loss/SRqqt1/detection_loss‚1.6885303867624162 /loss/FBt3/detection_loss‚1.9776879946390789 0loss/FRt2/assignment_loss‚1.2961630085681348 1loss/SRqqt2/assignment_loss‚1.234329896956502 1loss/FBt2/assignment_loss‚0.05976569258454709 1loss/SRqqt4/detection_loss‚2.4868354863590665¬¬ö`”¶‚(:‰ è¢êÔŠ­ð€ gpu.0.gpu‚44.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚158.469' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚39.61725 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ì$§¬Z·‚(:Ï è¢êÔ”âÅ‚ disk.rbd0.in‚0" proc.memory.rssMB‚ 2990.4921875 network.sent‚ 8.703833e+06 disk.md0p1.in‚ 0.17578125 disk.sdb.in‚ 0.08203125 disk.sda.in‚0.09375 disk.md0.in‚ 0.17578125 disk.sda1.out‚0 network.recv‚ 1.433248e+06 disk.dm-0.out‚50204.65234375# disk.nvme0n1.out‚23751.05078125 disk.sdb.out‚494.1357421875 disk.sda2.in‚0.09375 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.sda.out‚494.1357421875" disk.nvme1n1.out‚ 26453.6015625 disk.sdb2.in‚ 0.08203125$ memory_percent‚92.04034805297852) disk./.usagePercent‚75.85081559895777 disk.rbd0.out‚1.125 disk.sda1.in‚0 disk.nvme0n1.in‚ 603.27734375 disk.rbd2.out‚1.0625 disk.sdb2.out‚494.1357421875$ disk./.usageGB‚2712.792568206787 disk.md0p1.out‚ 476.15625 disk.sda2.out‚494.1357421875 disk.nvme1n1.in‚ 498.61328125 disk.rbd2.in‚0' proc.memory.availableMB‚ 1304.109375 cpu‚25.22522750927783 proc.cpu.threads‚17 disk.md0.out‚ 476.15625) proc.memory.percent‚18.25251579284668 disk.dm-0.in‚ 1101.890625äfèRd¸‚(:Ù ÷¢êÔÑý‚ÿ disk.sdb.out‚495.94091796875 disk.sda.out‚495.94091796875 disk.nvme1n1.in‚ 498.61328125 disk.rbd2.in‚0 cpu‚25.224490129558856 disk.sda1.in‚0 disk.dm-0.in‚ 1101.890625 network.sent‚ 8.798445e+06# disk.nvme0n1.out‚23751.62109375 disk.rbd0.in‚0! disk.sdb2.out‚495.94091796875 disk.md0p1.out‚ 477.890625$ memory_percent‚92.04037189483643 network.recv‚ 1.438369e+06 disk.sda.in‚0.09375 disk.sda2.in‚0.09375 disk.rbd0.out‚1.125 disk.sdb1.out‚0 disk.rbd2.out‚1.0625) proc.memory.percent‚18.25251579284668 disk.sdb2.in‚ 0.08203125 disk.dm-0.out‚ 50293.1796875) proc.memory.availableMB‚ 1304.10546875" proc.memory.rssMB‚ 2990.4921875) disk./.usagePercent‚75.85122538872217% disk./.usageGB‚2712.8072242736816 disk.nvme0n1.in‚ 603.27734375 disk.md0.out‚ 477.890625# disk.nvme1n1.out‚26541.55859375 disk.md0.in‚ 0.17578125 disk.sda1.out‚0 disk.md0p1.in‚ 0.17578125 disk.sdb.in‚ 0.08203125! disk.sda2.out‚495.94091796875 disk.sdb1.in‚0 proc.cpu.threads‚17Öâ’¹‚(:‡ ÷¢êÔ¾­òÿ gpu.0.gpu‚16.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚36.0 gpu.0.powerWatts‚114.408' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.602 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Àš¹»^º‚(:Ó †£êÔßžÒÿ disk.rbd2.out‚1.0625 disk.dm-0.in‚ 1101.890625 disk.nvme0n1.out‚ 23751.625" proc.memory.rssMB‚ 2990.4921875 disk.dm-0.out‚50519.16015625 disk.nvme1n1.in‚ 498.61328125 disk.sdb1.in‚0$ memory_percent‚92.04039573669434 proc.cpu.threads‚17 network.sent‚ 8.804804e+06$ disk./.usageGB‚2712.843536376953 disk.md0.out‚ 479.765625 disk.sda1.in‚0 disk.md0p1.out‚ 479.765625 disk.nvme0n1.in‚ 603.27734375 disk.sdb.out‚497.88330078125 disk.rbd0.out‚1.125 network.recv‚ 1.443081e+06 disk.rbd2.in‚0 disk.sda1.out‚0 disk.sda2.in‚0.09375! disk.sda2.out‚497.88330078125 disk.rbd0.in‚0 disk.sdb1.out‚0 cpu‚25.223822765127352 disk.sdb2.in‚ 0.08203125 disk.sda.in‚0.09375 disk.sda.out‚497.88330078125# disk.nvme1n1.out‚26767.53515625! disk.sdb2.out‚497.88330078125 disk.md0p1.in‚ 0.17578125( proc.memory.availableMB‚ 1304.1015625) proc.memory.percent‚18.25251579284668) disk./.usagePercent‚75.85224069032745 disk.md0.in‚ 0.17578125 disk.sdb.in‚ 0.08203125Љ’ “»‚(:ˆ †£êÔÅŸ‚€ gpu.0.gpu‚44.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚154.126' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚38.5315 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0)²&¢”¼‚(:‰ •£êÔ‚‘Àý gpu.0.gpu‚54.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚37.0 gpu.0.powerWatts‚160.305' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚40.07625 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0°n>™[½‚(:Ð •£êÔȽ¸ disk.nvme1n1.in‚ 498.61328125 network.sent‚ 8.812813e+06) disk./.usagePercent‚75.85235481709681 disk.sda1.out‚0 disk.md0.in‚ 0.17578125 disk.sda1.in‚0" proc.memory.rssMB‚ 2990.4921875 disk.dm-0.out‚ 50674.609375 disk.sda.in‚0.09375 disk.sda.out‚ 499.630859375 disk.sda2.out‚ 499.630859375( proc.memory.availableMB‚ 1304.1015625% disk./.usageGB‚2712.8476181030273# disk.nvme0n1.out‚23751.87109375 disk.sdb.in‚ 0.08203125 disk.rbd0.in‚0 disk.sdb1.in‚0 network.recv‚ 1.446196e+06 disk.md0p1.out‚ 481.4453125 disk.nvme0n1.in‚ 603.27734375 disk.sdb.out‚ 499.630859375 disk.sda2.in‚0.09375 disk.sdb2.in‚ 0.08203125 disk.md0p1.in‚ 0.17578125 disk.rbd0.out‚1.125$ memory_percent‚92.04039573669434 cpu‚25.22308837851611 proc.cpu.threads‚17 disk.rbd2.out‚1.0625# disk.nvme1n1.out‚26922.73828125 disk.sdb1.out‚0 disk.rbd2.in‚0 disk.sdb2.out‚ 499.630859375 disk.dm-0.in‚ 1101.890625 disk.md0.out‚ 481.4453125) proc.memory.percent‚18.25251579284668(øš`2¾- _wandbruntime‚4340  _runtime‚4340ª”RÇ¿Á & lr-AdamW/pg1‚0.0007515676972518015 & lr-AdamW/pg2‚0.0007515676972518015 trainer/global_step‚3699 % _timestamp‚1.7885147149394484e+09 _runtime‚4340.409882146 _step‚301\ÒûÌÀÆ & lr-AdamW/pg1‚0.0007515676972518015 & lr-AdamW/pg2‚0.0007515676972518015 trainer/global_step‚3699 % _timestamp‚1.7885147149394484e+09 _runtime‚4340.409882146 _step‚301­DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26088 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚69888982 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4210470078464 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177047040 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450780 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚308992938583552 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚304782468505088 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚308993855994368 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚92914646 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 398533632 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007241 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1212 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917410816 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407872 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚22308179 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚302150144458928 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 _step‚302 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 trainer/global_step‚3699 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚92888558 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚345377189997568 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚304782486508032 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955591168 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚76520418 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4395564708864 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚304782468505088 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4211369486336 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 _runtime‚4340.411117462 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚304782486508032 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚92888558 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚76520420 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4395799860224 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚297954455335856 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚92914646 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 169435066 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚308993855994368 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚345376930906112 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955591168 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚340981366197248 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚297954472375216 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4194774672292 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26090 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26088 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚308992938583552 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚47581415 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214665 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚47581413 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4195672083712 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚612 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚22307569 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚76520418 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚76520420 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407872 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 259091456 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚340981390137344 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1187 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚69889594 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4211369486336 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411420 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4210470078464 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚610 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360ˆ›À° ÃñO D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26090 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26088 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚308992938583552 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚47581415 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214665 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚47581413 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4195672083712 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚612 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚22307569 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚76520418 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚76520420 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407872 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 259091456 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚340981390137344 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1187 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚69889594 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4211369486336 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411420 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4210470078464 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚610 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410816 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26088 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚69888982 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 WEDeviceStatsMonitor.on_train_batch_start/act¡¤Give_bytes.small_pool.freed‚ 4210470078464 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177047040 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450780 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚308992938583552 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚304782468505088 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚308993855994368 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚92914646 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 398533632 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007241 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1212 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917410816 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407872 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚22308179 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚302150144458928 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 _step‚302 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 trainer/global_step‚3699 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚92888558 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚345377189997568 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚304782486508032 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955591168 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚76520418 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4395564708864 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚304782468505088 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4211369486336 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 _runtime‚4340.411117462 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚304782486508032 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚92888558 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚76520420 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4395799860224 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚297954455335856 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚92914646 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 169435066 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚308993855994368 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚345376930906112 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955591168 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚340981366197248 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚297954472375216 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4194774672292 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564®pßk2Ä- _wandbruntime‚4341  _runtime‚4341 þëóü&ÅöM SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899421184 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚614 _runtime‚4341.532853133 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚345467667167744 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚76540316 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚612 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚341070685417984 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 218360832 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 169479581 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4212482302976 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26114 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚309074599106560 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚69907609 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚92913149 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚309074599106560 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4195883333610 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917424128 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚76540318 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚341070709358080 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899421184 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 169453465 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚309073681682432 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚345467424866816 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚69906995 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1187 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955591168 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4211582881792 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177047040 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚47593791 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0n_ž'ÆûM BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4212482302976 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26114 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚309074599106560 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚69907609 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚92913149 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚309074599106560 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4195883333610 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917424128 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚76540318 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚341070709358080 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899421184 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 169453465 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚309073681682432 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚345467424866816 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚69906995 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1187 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955591168 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4211582881792 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177047040 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚47593791 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899421184 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚614 _runtime‚4341.532853133 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚345467667167744 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚76540316 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚612 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚341070685417984 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 218360832 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 169479581 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0¯¨˜!2Ç- _wandbruntime‚4341  _runtime‚4341 ëéœYÈÓ /loss/FRt1/assignment_loss‚3.107934985049935 2loss/SRqqt1/assignment_loss‚1.6444802744580167 1loss/SRqqt2/detection_loss‚0.8200782370092212 _step‚304 0loss/FRt4/assignment_loss‚3.2201425355413686 0loss/FBt1/assignment_loss‚0.5295529526192695 1loss/SRqqt1/detection_loss‚1.5929376726247826 .loss/FBt3/detection_loss‚3.186969041824341 _runtime‚4341.534819343 epoch‚3 /loss/FRt1/detection_loss‚1.9921843090808211 0loss/FRt2/assignment_loss‚1.4898733924264493 1loss/FBt3/assignment_loss‚0.20542876147846983 2loss/SRqqt2/assignment_loss‚0.8037711662287568 1loss/SRqqt3/assignment_loss‚1.349163349574575 .loss/FRt3/detection_loss‚2.560642560571432 /loss/FRt4/detection_loss‚3.2817062994708186 /loss/FBt1/detection_loss‚2.9619259323392595 /loss/FBt2/detection_loss‚0.4757843169000219 % _timestamp‚1.7885147160646482e+09 .loss/FBt4/detection_loss‚2.147163248062134 %loss/total_loss‚43.16832329825168 /loss/FRt2/detection_loss‚1.7427662186648534 /loss/FRt3/assignment_loss‚3.475492039322853 2loss/SRqqt4/assignment_loss‚1.8753126000656801 0loss/FBt2/assignment_loss‚0.2582524731389921 1loss/FBt4/assignment_loss‚0.17413074520827648 0loss/SRqqt4/detection_loss‚2.427672102170832 0loss/SRqqt3/detection_loss‚1.844958084420516 trainer/global_step‚3699+T·º^ÉØ /loss/FRt1/assignment_loss‚3.107934985049935 2loss/SRqqt1/assignment_loss‚1.6444802744580167 1loss/SRqqt2/detection_loss‚0.8200782370092212 _step‚304 0loss/FRt4/assignment_loss‚3.2201425355413686 0loss/FBt1/assignment_loss‚0.5295529526192695 1loss/SRqqt1/detection_loss‚1.5929376726247826 .loss/FBt3/detection_loss‚3.186969041824341 _runtime‚4341.534819343 epoch‚3 /loss/FRt1/detection_loss‚1.9921843090808211 0loss/FRt2/assignment_loss‚1.4898733924264493 1loss/FBt3/assignment_loss‚0.20542876147846983 2loss/SRqqt2/assignment_loss‚0.8037711662287568 1loss/SRqqt3/assignment_loss‚1.349163349574575 .loss/FRt3/detection_loss‚2.560642560571432 /loss/FRt4/detection_loss‚3.2817062994708186 /loss/FBt1/detection_loss‚2.9619259323392595 /loss/FBt2/detection_loss‚0.4757843169000219 % _timestamp‚1.7885147160646482e+09 .loss/FBt4/detection_loss‚2.147163248062134 %loss/total_loss‚43.16832329825168 /loss/FRt2/detection_loss‚1.7427662186648534 /loss/FRt3/assignment_loss‚3.475492039322853 2loss/SRqqt4/assignment_loss‚1.8753126000656801 0loss/FBt2/assignment_loss‚0.2582524731389921 1loss/FBt4/assignment_loss‚0.17413074520827648 0loss/SRqqt4/detection_loss‚2.427672102170832 0loss/SRqqt3/detection_loss‚1.844958084420516 trainer/global_step‚3699° Nèb“Ê‚(:ˆ ¤£êÔ¾¬úý gpu.0.gpu‚56.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚165.266' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚41.3165 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¡_#DSË‚(:È ¤£êÔ¾¦á€ disk.nvme0n1.out‚ 23751.90625( disk./.usagePercent‚75.8393453253334 disk.rbd0.out‚1.125 disk.sda1.in‚0 disk.dm-0.in‚ 1101.890625 disk.sdb.out‚ 500.9609375( proc.memory.availableMB‚ 1304.0703125 proc.cpu.threads‚17$ disk./.usageGB‚2712.382335662842 disk.rbd2.in‚0 disk.nvme0n1.in‚ 603.27734375) proc.memory.percent‚18.25251579284668 disk.sda2.in‚0.09375 disk.rbd0.in‚0 disk.md0.in‚ 0.17578125 disk.dm-0.out‚50795.99609375 disk.sdb.in‚ 0.08203125 disk.nvme1n1.in‚ 498.61328125 disk.sdb1.out‚0 network.sent‚ 8.816612e+06 disk.sda2.out‚ 500.9609375 disk.sdb1.in‚0 disk.rbd2.out‚1.0625 disk.sdb2.out‚ 500.9609375 cpu‚25.222274546343517 disk.sda.out‚ 500.9609375# disk.nvme1n1.out‚27044.08984375 disk.sdb2.in‚ 0.08203125 disk.sda1.out‚0 disk.md0p1.in‚ 0.17578125 disk.md0.out‚ 482.70703125$ memory_percent‚92.04058647155762" proc.memory.rssMB‚ 2990.4921875 network.recv‚ 1.450615e+06 disk.sda.in‚0.09375 disk.md0p1.out‚ 482.70703125ÍEOÌ‚(:Ä ³£êÔ¼ë›ÿ disk.sdb.out‚ 502.66015625 disk.nvme0n1.out‚23752.25 disk.rbd0.out‚1.125 disk.md0.out‚ 484.34375 network.sent‚ 8.925513e+06) disk./.usagePercent‚75.84319406397448 disk.sdb1.in‚0 disk.md0.in‚ 0.17578125 disk.sdb2.in‚ 0.08203125 disk.sda1.out‚0 disk.dm-0.in‚ 1101.890625 proc.cpu.threads‚17 disk.rbd0.in‚0 disk.rbd2.out‚1.0625 disk.dm-0.out‚ 50901.5390625) proc.memory.percent‚18.25253963470459 network.recv‚ 1.455633e+06 disk.md0p1.in‚ 0.17578125 disk.sda2.in‚0.09375 disk.nvme1n1.in‚ 498.61328125 disk.rbd2.in‚0) proc.memory.availableMB‚ 1303.42578125% disk./.usageGB‚2712.5199851989746 disk.md0p1.out‚ 484.34375 disk.sda2.out‚ 502.66015625 disk.sda1.in‚0# proc.memory.rssMB‚ 2990.49609375 cpu‚25.221647659327495 disk.sda.in‚0.09375" disk.nvme1n1.out‚ 27149.2890625 disk.sdb2.out‚ 502.66015625# memory_percent‚92.0445203781128 disk.nvme0n1.in‚ 603.27734375 disk.sda.out‚ 502.66015625 disk.sdb1.out‚0 disk.sdb.in‚ 0.08203125w¼kêÍ‚(:‰ ³£êÔæð¯ÿ gpu.0.gpu‚41.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚168.715' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.17875 gpu ™u.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0_(Aڒ΂(:‡ £êÔ´ÜÙý gpu.0.gpu‚40.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚166.548' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚41.637 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¬¤fÏ‚(:Û £êÔͱ‹ÿ) proc.memory.percent‚18.25253963470459 cpu‚25.221007952598264 disk.sdb1.out‚0% disk./.usageGB‚2712.8462829589844 disk.rbd2.in‚0 disk.rbd2.out‚1.0625 disk.sdb.out‚504.49951171875 disk.sdb1.in‚0 disk.md0.out‚ 486.12890625' proc.memory.availableMB‚ 1302.984375! disk.sdb2.out‚504.49951171875 disk.sda2.in‚0.09375 disk.rbd0.out‚1.125$ memory_percent‚92.04721450805664 network.recv‚ 1.460399e+06 disk.dm-0.in‚ 1101.890625 disk.sdb2.in‚ 0.08203125 disk.dm-0.out‚51074.83203125 disk.md0p1.in‚ 0.17578125 disk.nvme1n1.in‚ 498.61328125 disk.md0p1.out‚ 486.12890625# disk.nvme0n1.out‚23752.28515625 disk.sda.out‚504.49951171875) disk./.usagePercent‚75.85231748591058 disk.sda1.out‚0 disk.nvme0n1.in‚ 603.27734375# proc.memory.rssMB‚ 2990.49609375 proc.cpu.threads‚17 disk.sda1.in‚0 disk.sda.in‚0.09375! disk.nvme1n1.out‚ 27322.546875 disk.md0.in‚ 0.17578125 network.sent‚ 8.931801e+06 disk.sdb.in‚ 0.08203125! disk.sda2.out‚504.49951171875 disk.rbd0.in‚0$ Z’Ђ(:‡ Ñ£êÔó¹Øÿ gpu.0.gpu‚47.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚161.864' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚40.466 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0°ÍÔ6WÑ‚(:Ì Ñ£êÔààœ€ disk.md0.out‚487.75 disk.rbd2.in‚0 disk.sdb2.in‚ 0.08203125# proc.memory.rssMB‚ 2990.49609375 network.sent‚ 8.934855e+06 disk.sda.in‚0.09375 disk.sda1.in‚0 disk.sdb.out‚506.1689453125 proc.cpu.threads‚17 network.recv‚ 1.463887e+06 disk.sda2.out‚506.1689453125 disk.rbd0.in‚0 disk.sdb2.out‚506.1689453125 disk.md0p1.out‚487.75 disk.sda2.in‚0.09375 disk.sdb1.in‚0 disk.dm-0.in‚ 1101.890625 disk.dm-0.out‚51257.01171875 disk.sdb.in‚ 0.08203125$ memory_percent‚92.04576015472412) proc.memory.availableMB‚ 1303.22265625 disk.nvme0n1.in‚ 603.27734375 disk.sda.out‚506.1689453125 disk.md0.in‚ 0.17578125 disk.sda1.out‚0) proc.memory.percent‚18.25253963470459 cpu‚25.22035762551889) disk./.usagePercent‚75.85410628969478# disk.nvme1n1.out‚27504.53515625 disk.rbd2.out‚1.0625 disk.rbd0.out‚1.125 disk.md0p1.in‚ 0.17578125" disk.nvme0n1.out‚ 23752.4765625$ disk./.usageGB‚2712.910259246826 disk.nvme1n1.in‚ 498.61328125 disk.sdb1.out‚0%³2Ò- _wandbruntime‚4397  _runtime‚4397iæ#ÇÓÁ & lr-AdamW/pg1‚0.0007474570295478668 & lr-AdamW/pg2‚0.0007474570295478668 trainer/global_step‚3749 % _timestamp‚1.7885147724058151e+09 _runtime‚4397.876125231 _step‚305A+iÌÔÆ & lr-AdamW/pg2‚0.0007474570295478668 trainer/global_step‚3749 % _timestamp‚1.7885147724058151e+09 _runtime‚4397.876125231 _step‚305 & lr-AdamW/pg1‚0.0007474570295478668±àäÙ 2Õ- _wandbruntime‚4397  _runtime‚4397i¨ò'ÖìO QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410816 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚345447327234048 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚313030107941376 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4266124658688 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚308763983282688 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚301853477081616 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4251118740820 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚313031025352192 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚349900811832320 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407872 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚94121460 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407872 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4266124658688 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4453715555328 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4453484598272 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955591168 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4267024066560 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚94147548 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚306104595822436 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚308763983282688 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚94121460 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚313030107941376 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1187 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚617 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚77515320 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚48200213 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚22591533 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 398533632 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450780 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚22590916 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955591168 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚77515318 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚48200215 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 % _timestamp‚1.7885147724065328e+09 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚313031025352192 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917410816 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0ÿIdp÷'×ñO UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚308764001285632 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26088 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _runtime‚4397.879999169 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177047040 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 171662868 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚301853460042256 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚94121460 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚313030107941376 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1187 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚617 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚77515320 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚48200213 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚22591533 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 398533632 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450780 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚22590916 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955591168 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚77515318 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚48200215 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 % _timestamp‚1.7885147724065328e+09 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚313031025352192 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917410816 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410816 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚345447327234048 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚313030107941376 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4266124658688 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚308763983282688 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚301853477081616 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4251118740820 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚313031025352192 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚349900811832320 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407872 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚94121460 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407872 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4266124658688 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4453715555328 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4453484598272 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955591168 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4267024066560 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚94147548 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚306104595822436 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚308763983282688 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 JDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 171707383 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4252227402346 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚77535216 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177047040 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚77535218 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚308843631581184 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 171681267 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚70809085 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚596 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚22597090 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚48212589 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4251329990718 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚22596496 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚301931440136384 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚48212591 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1187 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 171707383 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4252227402346 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚77535216 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177047040 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚77535218 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚308843631581184 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 171681267 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚70809085 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚596 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚22597090 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚48212589 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4251329990718 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚22596496 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚301931440136384 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚48212591 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1187 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 PDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚317067277299200 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410816 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚317068194710016 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚354424731234304 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 398533632 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚310058132735164 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚317067277299200 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚312745516063232 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚22874296 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4321779238912 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚71692691 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 254897152 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4511673920512 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4321779238912 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚78510220 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚349913288270848 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚305752464748656 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214665 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4322678646784 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1187 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚354424986131456 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚349913312210944 _runtime‚4455.328620372 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚312745498060288 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4322678646784 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚48819015 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚305752481788016 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 trainer/global_step‚3799 $ _timestamp‚1.788514829858218e+09 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4306565397928 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚78510220 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚312745516063232 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411420 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚78510218 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917410816 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚95380450 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚310059047185944 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450780 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚620 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚22873678 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007241 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 _step‚310 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26088 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚95354362 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955591168 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407872 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 173890670 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚317067277299200 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410816 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚317068194710016 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚354424731234304 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 398533632 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚310058132735164 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚317067277299200 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚312745516063232 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚22874296 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4321779238912 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚71692691 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 254897152 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4511673920512 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4321779238912 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚78510220 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚349913288270848 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚305752464748656 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214665 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4322678646784 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1187 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚354424986131456 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚349913312210944 _runtime‚4455.328620372 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚312745498060288 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4322678646784 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚48819015 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚305752481788016 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 trainer/global_step‚3799 $ _timestamp‚1.788514829858218e+09 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4306565397928 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚78510220 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚312745516063232 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411420 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚78510218 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917410816 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚95380450 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚310059047185944 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450780 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564¶ÿ¥Ÿ¹2ì- _wandbruntime‚4456  _runtime‚4456Ü20ü&íöM UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4323791463424 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899421184 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚312825128355840 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚48831391 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚78530116 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899421184 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 173935185 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚354515473307136 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚350002607491584 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚310137221490610 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚305830444842784 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 173909069 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚317148020398080 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚305830461882144 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4323791463424 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 173935185 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177047040 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚595 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚78530118 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚593 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4306776647826 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚71711311 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4322892042240 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚71710716 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚48831389 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 233912320 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4307674059454 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚312825146358784¼ÀǸ'îûM UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚78530118 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 173909069 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚22879920 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 _step‚311 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚317148937822208 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899421184 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 173935185 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚354515473307136 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚350002607491584 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚310137221490610 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚305830444842784 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 173909069 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚317148020398080 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚305830461882144 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4323791463424 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 173935185 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177047040 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚595 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚78530118 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚593 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4306776647826 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚71711311 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4322892042240 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚71710716 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚48831389 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 233912320 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4307674059454 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚312825146358784 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4323791463424 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899421184 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚312825128355840 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚48831391 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚78530116 EΔÏ2û- _wandbruntime‚4507  _runtime‚4507N%ÞÙò'üìO E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚79505120 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚358948922795520 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007241 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚79505118 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚309651486494416 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 176092382 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚23157513 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 246508544 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚314012584098672 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚79505120 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚316727012837888 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚609 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 398533632 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚316727030840832 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4362012055036 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411420 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _runtime‚4507.080899375 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚321104446657024 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚321104446657024 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚49437815 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 176118472 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1187 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4361114643616 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26090 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚96587264 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407872 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚72594717 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214665 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚321105364067840 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4569673487872 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450780 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177047040 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚79505118 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26088 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407872 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚96587264 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚358949169304064 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 trainer/global_step‚3849 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚96613352 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4378333227008 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚611 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚316727030840832 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0Tp[É÷'ýñO WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4362012055036 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411420 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _runtime‚4507.080899375 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚321104446657024 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚321104446657024 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚49437815 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 176118472 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1187 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4361114643616 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26090 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚96587264 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407872 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚72594717 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214665 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚321105364067840 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4569673487872 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450780 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177047040 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚79505118 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26088 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407872 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚96587264 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚358949169304064 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 trainer/global_step‚3849 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚96613352 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4378333227008 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚611 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚316727030840832 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚79505120 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚358948922795520 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007241 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚79505118 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚309651486494416 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 176092382 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚23157513 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 246508544 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚314012584098672 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚79505120 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚316727012837888 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚609 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 398533632 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚316727030840832º:ü2þ- _wandbruntime‚4508  _runtime‚4508A§ÕX| ÿõM M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4378546622464 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31686 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _step‚315 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚618 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚79525018 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411628 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚96637969 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26114 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚359039388997632 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4570820469248 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚316806661136384 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4379446043648 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚321186107180032 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚72612674 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 398533632 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚316806643133440 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚72613292 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚314091672854118 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4362223304934 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚4508.086680487 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 176162987 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4379446043648 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚79525018 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚321185189755904 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899421184 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚316806643133440 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚321186107180032 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚359039645978624 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4363120716562 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚309729466588544 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚49450191 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 256980992 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚96611855 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4378546622464 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚96637969 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 233040896 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 176162987 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 $ _timestamp‚1.788514882615331e+09 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917424128 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 176136871 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚23162485 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4571053510144 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26114 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚309729449549184 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 176136871 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450988 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007241—B3'€úM ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚316806661136384 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4379446043648 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚321186107180032 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚72612674 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 398533632 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚316806643133440 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚72613292 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚314091672854118 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4362223304934 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚4508.086680487 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 176162987 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4379446043648 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚79525018 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚321185189755904 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899421184 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚316806643133440 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚321186107180032 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚359039645978624 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4363120716562 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚309729466588544 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚49450191 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 256980992 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚96611855 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4378546622464 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚96637969 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 233040896 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 176162987 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 $ _timestamp‚1.788514882615331e+09 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917424128 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 176136871 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚23162485 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4571053510144 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26114 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚309729449549184 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 176136871 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450988 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007241 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4378546622464 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31686 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _step‚315 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚618 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚79525018 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411628 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚96637969 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26114 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚359039388997632 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4570820469248 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0»bº|2- _wandbruntime‚4508  _runtime‚4508pØ–ï^‚Ø 0loss/FRt2/assignment_loss‚1.7266396755973499 0loss/FRt3/assignment_loss‚2.7469092338692906 2loss/SRqqt3/assignment_loss‚1.3939710723236203 1loss/FBt1/assignment_loss‚0.15269963314494817 0loss/FBt4/assignment_loss‚0.2515730119269144 &loss/total_loss‚42.106433052871104 _runtime‚4508.087763272 2loss/SRqqt1/assignment_loss‚1.3123267422006888 0loss/SRqqt4/detection_loss‚2.491605347022414 2loss/SRqqt4/assignment_loss‚1.8812671322375536 .loss/FRt4/detection_loss‚3.288103678226471 /loss/FBt1/detection_loss‚3.1289003118872643 .loss/FBt3/detection_loss‚2.556400554520743 _step‚316 /loss/FRt1/assignment_loss‚3.010085765611042 1loss/SRqqt2/assignment_loss‚1.090906430148717 1loss/FBt2/assignment_loss‚0.07362163092286736 1loss/FBt3/assignment_loss‚0.02237618855977936 /loss/FRt3/detection_loss‚2.1960749842372596 1loss/SRqqt3/detection_loss‚1.6561280995607377 /loss/FRt2/detection_loss‚1.9213648550212383 trainer/global_step‚3849 /loss/FRt1/detection_loss‚2.3187553618442465 /loss/FBt2/detection_loss‚0.5960536653641612 /loss/FBt4/detection_loss‚2.2260974889451806 0loss/FRt4/assignment_loss‚3.5348451566696166 1loss/SRqqt1/detection_loss‚1.5384614995934747 1loss/SRqqt2/detection_loss‚0.9912655334355254 epoch‚3 % _timestamp‚1.7885148826176486e+09Y^ScƒÝ 1loss/FBt1/assignment_loss‚0.15269963314494817 0loss/FBt4/assignment_loss‚0.2515730119269144 &loss/total_loss‚42.106433052871104 _runtime‚4508.087763272 2loss/SRqqt1/assignment_loss‚1.3123267422006888 0loss/SRqqt4/detection_loss‚2.491605347022414 2loss/SRqqt4/assignment_loss‚1.8812671322375536 .loss/FRt4/detection_loss‚3.288103678226471 /loss/FBt1/detection_loss‚3.1289003118872643 .loss/FBt3/detection_loss‚2.556400554520743 _step‚316 /loss/FRt1/assignment_loss‚3.010085765611042 1loss/SRqqt2/assignment_loss‚1.090906430148717 1loss/FBt2/assignment_loss‚0.07362163092286736 1loss/FBt3/assignment_loss‚0.02237618855977936 /loss/FRt3/detection_loss‚2.1960749842372596 1loss/SRqqt3/detection_loss‚1.6561280995607377 /loss/FRt2/detection_loss‚1.9213648550212383 trainer/global_step‚3849 /loss/FRt1/detection_loss‚2.3187553618442465 /loss/FBt2/detection_loss‚0.5960536653641612 /loss/FBt4/detection_loss‚2.2260974889451806 0loss/FRt4/assignment_loss‚3.5348451566696166 1loss/SRqqt1/detection_loss‚1.5384614995934747 1loss/SRqqt2/detection_loss‚0.9912655334355254 epoch‚3 % _timestamp‚1.7885148826176486e+09 0loss/FRt2/assignment_loss‚1.7266396755973499 0loss/FRt3/assignment_loss‚2.7469092338692906 2loss/SRqqt3/assignment_loss‚1.3939710723236203¼ñÀ_=”„‚(:‰ ɤêÔÈÝÅ gpu.0.gpu‚10.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚37.0 gpu.0.powerWatts‚114.131' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.53275 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0NÏi—_…‚(:Ô ɤêÔçýŸ‚$ disk./.usageGB‚2712.341667175293 disk.rbd0.in‚0 disk.dm-0.in‚ 1101.890625) proc.memory.percent‚18.25253963470459 disk.nvme0n1.in‚ 603.27734375 disk.sdb1.out‚0 disk.sda.in‚0.09375 disk.sda2.in‚0.09375" disk.nvme1n1.out‚ 28781.2265625 disk.md0.out‚ 500.71875 disk.sda1.in‚0 disk.md0p1.in‚ 0.17578125 network.sent‚ 9.18421e+06 disk.sdb.out‚519.62158203125 disk.md0.in‚ 0.17578125 disk.md0p1.out‚ 500.71875$ memory_percent‚92.04010963439941 proc.cpu.threads‚17# disk.nvme0n1.out‚23830.30859375 disk.sdb1.in‚0 disk.rbd2.in‚0! disk.sdb2.out‚519.62158203125 disk.dm-0.out‚52611.53515625 network.recv‚ 1.499467e+06 disk.nvme1n1.in‚ 498.61328125 disk.rbd2.out‚1.0625( proc.memory.availableMB‚ 1304.1484375# proc.memory.rssMB‚ 2990.49609375 cpu‚25.21517077646918 disk.sdb2.in‚ 0.08203125 disk.rbd0.out‚1.125) disk./.usagePercent‚75.83820821740045 disk.sdb.in‚ 0.08203125 disk.sda.out‚519.62158203125! disk.sda2.out‚519.62158203125 disk.sda1.out‚0"Wü‘†‚(:† ؤêÔ¹ù¶þ gpu.0.gpu‚33.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚37.0 gpu.0.powerWatts‚114.74' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.685 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¬”qb‡‚(:× ؤêÔɧÒþ disk.sdb.in‚ 0.08203125 disk.sda1.in‚0) proc.memory.availableMB‚ 1303.75390625) proc.memory.percent‚18.25253963470459 disk.sda1.out‚0 disk.md0p1.out‚ 502.546875! disk.sda2.out‚521.51708984375# disk.nvme1n1.out‚28852.13671875 disk.rbd2.in‚0 disk.dm-0.out‚52713.75390625 disk.sda.in‚0.09375 disk.sda2.in‚0.09375! disk.sdb2.out‚521.51708984375$ memory_percent‚92.04251766204834$ disk./.usageGB‚2712.410446166992 disk.sdb.out‚521.51708984375 disk.nvme1n1.in‚ 498.61328125 network.recv‚ 1.504866e+06 disk.rbd0.in‚0 proc.cpu.threads‚17 disk.md0p1.in‚ 0.17578125 disk.sda.out‚521.51708984375 disk.rbd0.out‚1.125 disk.md0.out‚ 502.546875 disk.rbd2.out‚1.0625 disk.sdb2.in‚ 0.08203125 disk.dm-0.in‚ 1101.890625 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.md0.in‚ 0.17578125# proc.memory.rssMB‚ 2990.49609375 cpu‚25.21444923461848" disk.nvme0n1.out‚ 23861.6171875 network.sent‚ 9.291605e+06( disk./.usagePercent‚75.8401313067946 disk.nvme0n1.in‚ 603.27734375êúµ(”ˆ‚(:‰ ç¤êÔ½’íÿ gpu.0.gpu‚11.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚37.0 gpu.0.powerWatts‚115.025' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.75625 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0…Â/}a‰‚(:Ö ç¤êÔåôˆ€ disk.sdb.in‚ 0.08203125 disk.md0.in‚ 0.17578125' proc.memory.availableMB‚ 653.8046875 cpu‚25.213236660186453 disk.sda2.in‚0.09375! disk.sda2.out‚523.23486328125 disk.rbd2.in‚0 disk.sdb2.in‚ 0.08203125 disk.sda1.out‚0 disk.md0p1.in‚ 0.17578125 disk.sdb.out‚523.23486328125 disk.rbd0.in‚0" disk.nvme1n1.out‚ 28856.7421875! disk.sdb2.out‚523.23486328125 disk.dm-0.in‚ 1101.890625 disk.sda.out‚523.23486328125# proc.memory.rssMB‚ 2990.49609375 network.sent‚ 9.299649e+06 disk.sdb1.in‚0 disk.dm-0.out‚ 52805.328125$ memory_percent‚96.00949287414551) disk./.usagePercent‚75.84718402116009 disk.rbd2.out‚1.0625 disk.nvme0n1.in‚ 603.27734375 disk.sda.in‚0.09375) proc.memory.percent‚18.25253963470459 network.recv‚ 1.508805e+06$ disk./.usageGB‚2712.662685394287 disk.rbd0.out‚1.125 disk.md0.out‚ 504.1953125 disk.sda1.in‚0 disk.md0p1.out‚ 504.1953125" disk.nvme0n1.out‚ 23948.5859375 proc.cpu.threads‚17 disk.nvme1n1.in‚ 498.61328125 disk.sdb1.out‚0¬¦¯Ý”Š‚(:‰ ö¤êÔéƒÌÿ gpu.0.gpu‚54.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚166.825' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚41.70625 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0U÷»``‹‚(:Õ ö¤êÔÔÝü€ network.sent‚ 9.303766e+06% disk./.usageGB‚2713.1003646850586 disk.rbd0.out‚1.125 disk.sdb1.in‚0$ memory_percent‚96.01273536682129 disk.sda1.out‚0 disk.md0.out‚ 505.6640625# proc.memory.rssMB‚ 2990.49609375) disk./.usagePercent‚75.85942171731307 disk.rbd2.out‚1.0625 disk.sdb2.in‚ 0.08203125 disk.sda1.in‚0 disk.nvme0n1.in‚ 603.27734375 disk.sda2.in‚0.09375 disk.sda.in‚0.09375 disk.nvme1n1.in‚ 498.61328125# disk.nvme1n1.out‚29006.90234375 disk.dm-0.in‚ 1101.890625 network.recv‚ 1.513514e+06 disk.rbd2.in‚0 disk.sdb1.out‚0) proc.memory.percent‚18.25253963470459 disk.sdb.out‚524.7744140625 disk.sda2.out‚524.7744140625 disk.md0.in‚ 0.17578125' proc.memory.availableMB‚ 653.2734375 disk.dm-0.out‚53070.94140625 disk.sdb.in‚ 0.08203125 cpu‚25.21258344756918 proc.cpu.threads‚17 disk.sdb2.out‚524.7744140625" disk.nvme0n1.out‚ 24064.0390625 disk.sda.out‚524.7744140625 disk.rbd0.in‚0 disk.md0p1.in‚ 0.17578125 disk.md0p1.out‚ 505.6640625ð¹Ö#2Œ- _wandbruntime‚4561  _runtime‚4561( GÇÁ & lr-AdamW/pg1‚0.0007347301390788157 & lr-AdamW/pg2‚0.0007347301390788157 trainer/global_step‚3899 % _timestamp‚1.7885149358709972e+09 _runtime‚4561.341398787 _step‚317újéÌŽÆ & lr-AdamW/pg1‚0.0007347301390788157 & lr-AdamW/pg2‚0.0007347301390788157 trainer/global_step‚3899 % _timestamp‚1.7885149358709972e+09 _runtime‚4561.341398787 _step‚317½„ÎOi2- _wandbruntime‚4561  _runtime‚4561uÉIò'ìO PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚97820166 _step‚318 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚325142533425664 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410816 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4416561300724 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 178346274 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚97846254 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚363472837712384 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚358845234284544 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚325141616014848 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚325141616014848 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚623 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚358845210344448 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚80500020 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407872 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚23442373 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4433088399360 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚73498363 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4433987807232 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚50056613 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 230957056 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚320708527615488 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚80500018 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177047040 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 254897152 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4433088399360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1187 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26088 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26088 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1212 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚363473092609536 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚317967035462180 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007241 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚320708545618432 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 398533632 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26090 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚97846254 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411420 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407872 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚80500020 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚80500018 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚320708545618432%ê4Q‘ñO A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177047040 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 254897152 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4433088399360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.cuüJ‘Ž¦"rrent‚18002944 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1187 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26088 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26088 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1212 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚363473092609536 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚317967035462180 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007241 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚320708545618432 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 398533632 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26090 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚97846254 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411420 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407872 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚80500020 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚80500018 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚320708545618432 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚97820166 _step‚318 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚325142533425664 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410816 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4416561300724 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 178346274 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚97846254 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚363472837712384 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚358845234284544 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚325141616014848 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚325141616014848 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚623 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚358845210344448 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚80500020 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407872 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚23442373 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4433088399360 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚73498363 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4433987807232 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚50056613 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 230957056 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚320708527615488 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚80500018 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1¾ ú¹2’- _wandbruntime‚4562  _runtime‚4562-$›Lû&“õM NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚97844757 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 178364673 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚80519916 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4434201202688 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899421184 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚97870871 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚97844757 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 228846592 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚313628454255584 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚325223276537856 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚23448036 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚358934529565184 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4417669962042 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899421184 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚358934553505280 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4628783976960 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚313628471294944 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚318046124217626 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4418567373670 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450988 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 _step‚319 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚50068991 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚325223276537856 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4435100623872 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007241 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚320788175913984 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26114 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 252786688 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚73516409 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚50068989 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 $ _timestamp‚1.788514936878015e+09 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚320788157911040 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚73517027 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4435100623872 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚363563566328832 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚618 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚320788157911040 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4434201202688 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚616 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚325222359113728 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚4562.349008366 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31686 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚318047038668614 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 398533632 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177047040 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955591168 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚320788175913984 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411628 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚80519916 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 178390789 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ÚË'”úM UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899421184 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚97870871 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚97844757 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 228846592 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚313628454255584 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚325223276537856 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚23448036 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚358934529565184 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4417669962042 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899421184 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚358934553505280 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4628783976960 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚313628471294944 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚318046124217626 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4418567373670 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450988 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 _step‚319 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚50068991 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚325223276537856 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4435100623872 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007241 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚320788175913984 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26114 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 252786688 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚73516409 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚50068989 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 $ _timestamp‚1.788514936878015e+09 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚320788157911040 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚73517027 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4435100623872 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚363563566328832 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚618 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚320788157911040 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4434201202688 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚616 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚325222359113728 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚4562.349008366 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31686 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚318047038668614 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 398533632 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177047040 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955591168 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚320788175913984 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411628 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚80519916 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 178390789 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚97844757 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 178364673 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚80519916 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4434201202688¿ð;v£2•- _wandbruntime‚4562  _runtime‚45626xw´Y–Ó .loss/FBt1/detection_loss‚2.659909466902415 2loss/SRqqt2/assignment_loss‚0.9741930169567689 /loss/SRqqt4/detection_loss‚2.94985415365385 /loss/FBt3/detection_loss‚2.6685113668441773 trainer/global_step‚3899 _runtime‚4562.350516633 _step‚320 0loss/FRt1/assignment_loss‚2.3513781864468646 1loss/FBt3/assignment_loss‚0.08409406634164043 /loss/FRt1/detection_loss‚1.9929708730229048 .loss/FRt3/detection_loss‚2.358377773401349 .loss/FRt4/detection_loss‚3.293683041696963 1loss/SRqqt3/detection_loss‚1.7583147462370783 epoch‚3 0loss/FRt3/assignment_loss‚2.6867689795272294 2loss/SRqqt1/assignment_loss‚1.5124054619270126 /loss/FBt1/assignment_loss‚0.531011499056954 1loss/FBt2/assignment_loss‚0.06294867398995821 1loss/SRqqt2/detection_loss‚0.9479936798339459 /loss/FBt4/detection_loss‚2.9015629953808255 2loss/SRqqt4/assignment_loss‚1.1508731991052628 0loss/FBt4/assignment_loss‚0.2004013174899026 $ _timestamp‚1.788514936880402e+09 0loss/SRqqt1/detection_loss‚1.833140843343444 %loss/total_loss‚41.49251381890987 2loss/SRqqt3/assignment_loss‚0.9779487869074178 /loss/FBt2/detection_loss‚0.6869032495680635 0loss/FRt2/assignment_loss‚1.4439305981000266 0loss/FRt4/assignment_loss‚3.7191433984300364 /loss/FRt2/detection_loss‚1.7461944447457791˜;@{^—Ø 0loss/SRqqt1/detection_loss‚1.833140843343444 %loss/total_loss‚41.49251381890987 2loss/SRqqt3/assignment_loss‚0.9779487869074178 /loss/FBt2/detection_loss‚0.6869032495680635 0loss/FRt2/assignment_loss‚1.4439305981000266 0loss/FRt4/assignment_loss‚3.7191433984300364 /loss/FRt2/detection_loss‚1.7461944447457791 .loss/FBt1/detection_loss‚2.659909466902415 2loss/SRqqt2/assignment_loss‚0.9741930169567689 /loss/SRqqt4/detection_loss‚2.94985415365385 /loss/FBt3/detection_loss‚2.6685113668441773 trainer/global_step‚3899 _runtime‚4562.350516633 _step‚320 0loss/FRt1/assignment_loss‚2.3513781864468646 1loss/FBt3/assignment_loss‚0.08409406634164043 /loss/FRt1/detection_loss‚1.9929708730229048 .loss/FRt3/detection_loss‚2.358377773401349 .loss/FRt4/detection_loss‚3.293683041696963 1loss/SRqqt3/detection_loss‚1.7583147462370783 epoch‚3 0loss/FRt3/assignment_loss‚2.6867689795272294 2loss/SRqqt1/assignment_loss‚1.5124054619270126 /loss/FBt1/assignment_loss‚0.531011499056954 1loss/FBt2/assignment_loss‚0.06294867398995821 1loss/SRqqt2/detection_loss‚0.9479936798339459 /loss/FBt4/detection_loss‚2.9015629953808255 2loss/SRqqt4/assignment_loss‚1.1508731991052628 0loss/FBt4/assignment_loss‚0.2004013174899026 $ _timestamp‚1.788514936880402e+09À]]ÑË”˜‚(:‰ …¥êÔõ‰®ÿ gpu.0.gpu‚55.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚170.891' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.72275 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0çð q™‚(:Ò …¥êÔ–îñ( proc.memory.availableMB‚ 653.74609375 proc.cpu.threads‚17 disk.sda2.in‚0.09375 network.sent‚ 9.321848e+06 disk.dm-0.in‚ 1101.890625 disk.md0p1.out‚ 507.23828125 disk.sdb.out‚ 526.392578125 disk.sda.in‚0.09375 disk.sda.out‚ 526.392578125 disk.rbd0.in‚0% disk./.usageGB‚2713.3498725891113 disk.md0.out‚ 507.23828125 disk.rbd2.out‚1.0625 disk.nvme0n1.in‚ 603.27734375 disk.sdb.in‚ 0.08203125$ memory_percent‚96.00985050201416 disk.rbd2.in‚0 disk.sdb2.in‚ 0.08203125 disk.sda1.out‚0# proc.memory.rssMB‚ 2990.49609375Ê#O³ì) proc.memory.percent‚18.25253963470459 cpu‚25.211933918280643) disk./.usagePercent‚75.86639806273755 disk.md0.in‚ 0.17578125 disk.sda1.in‚0 disk.dm-0.out‚ 53295.015625 disk.sda2.out‚ 526.392578125 disk.rbd0.out‚1.125 disk.nvme1n1.in‚ 498.61328125" disk.nvme1n1.out‚ 29115.3203125 disk.sdb2.out‚ 526.392578125 disk.md0p1.in‚ 0.17578125" disk.nvme0n1.out‚ 24179.6953125 disk.sdb1.in‚0 network.recv‚ 1.517006e+06 disk.sdb1.out‚0„ˆº9”š‚(:‰ ”¥êÔ˜¶òý gpu.0.gpu‚38.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚171.501' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.87525 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0­S׃a›‚(:Ö ”¥êÔÜûœ€ disk.md0.in‚ 0.17578125 network.sent‚ 9.418515e+06 disk.rbd2.in‚0 disk.sdb2.out‚528.1884765625# proc.memory.rssMB‚ 2990.49609375 network.recv‚ 1.522101e+06 disk.sdb.in‚ 0.08203125 disk.sdb1.out‚0 disk.sda1.in‚0( proc.memory.availableMB‚ 653.69921875 disk.sdb1.in‚0 disk.md0.out‚ 508.9765625 disk.sdb2.in‚ 0.08203125 disk.sda.out‚528.1884765625 disk.rbd0.out‚1.125 disk.nvme1n1.in‚ 498.61328125 disk.dm-0.in‚ 1101.890625 disk.md0p1.in‚ 0.17578125 cpu‚25.211372515611544) disk./.usagePercent‚75.86639646282958% disk./.usageGB‚2713.3498153686523 disk.sda.in‚0.09375 disk.rbd2.out‚1.0625 disk.md0p1.out‚ 508.9765625 proc.cpu.threads‚17 disk.rbd0.in‚0 disk.sda1.out‚0 disk.dm-0.out‚ 53523.734375$ memory_percent‚96.01013660430908 disk.nvme0n1.in‚ 603.27734375 disk.sdb.out‚528.1884765625 disk.sda2.in‚0.09375) proc.memory.percent‚18.25253963470459# disk.nvme0n1.out‚24266.31640625 disk.sda2.out‚528.1884765625# disk.nvme1n1.out‚29257.41796875Ûè1R”œ‚(:‰ £¥êÔßÊÙ€ gpu.0.gpu‚42.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚162.813' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚40.70325 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0áfØd‚(:Ù £¥êԱ꣄ disk.sdb1.in‚0( proc.memory.availableMB‚ 653.33984375 disk.sda.out‚529.77685546875 disk.nvme1n1.in‚ 498.61328125 disk.rbd2.out‚1.0625 disk.sdb2.in‚ 0.08203125) proc.memory.percent‚18.25253963470459 disk.sda2.in‚0.09375 disk.sdb1.out‚0 disk.md0.out‚ 510.4921875 disk.md0p1.out‚ 510.4921875 disk.sdb.in‚ 0.08203125! disk.sdb2.out‚529.77685546875 proc.cpu.threads‚17 network.sent‚ 9.422426e+06 disk.rbd0.out‚1.125 disk.dm-0.in‚ 1101.890625 disk.sdb.out‚529.77685546875 disk.md0.in‚ 0.17578125 disk.sda1.out‚0 disk.dm-0.out‚53760.85546875# disk.nvme0n1.out‚24353.18359375 cpu‚25.21075023515604 disk.sda.in‚0.09375 disk.sda1.in‚0 disk.md0p1.in‚ 0.17578125! disk.sda2.out‚529.77685546875 disk.rbd0.in‚0! disk.nvme1n1.out‚ 29407.671875 disk.rbd2.in‚0 disk.nvme0n1.in‚ 603.27734375$ memory_percent‚96.01233005523682# proc.memory.rssMB‚ 2990.49609375 network.recv‚ 1.526232e+06) disk./.usagePercent‚75.86639646282958% disk./.usageGB‚2713.3498153686523Œp2ž- _wandbruntime‚4617  _runtime‚4617— ¾ÆŸÀ trainer/global_step‚3949 % _timestamp‚1.7885149924161904e+09 _runtime‚ 4617.88657563 _step‚321 & lr-AdamW/pg1‚0.0007303591601181872 & lr-AdamW/pg2‚0.0007303591601181872k|)XË Å _step‚321 & lr-AdamW/pg1‚0.0007303591601181872 & lr-AdamW/pg2‚0.0007303591601181872 trainer/global_step‚3949 % _timestamp‚1.7885149924161904e+09 _runtime‚ 4617.88657563Á¯ÈLÇ2¡- _wandbruntime‚4617  _runtime‚4617/A”ò'¢ìO R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚329178785372672 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4489642387456 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚329178785372672 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26088 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚329179702783488 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 233054208 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955591168 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _step‚322 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26090 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚591 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1187 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410816 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407872 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚74401593 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚81494918 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚23726771 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚317449478867856 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚23726180 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4472007957832 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚99053068 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚99053068 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917410816 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚593 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 398533632 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚3949 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 180547986 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4488742979584 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚81494920 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4685691100672 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 % _timestamp‚1.7885149924173126e+09 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚367997119476224 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450780 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚324690060396032 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚324690042393088 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4489642387456 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚329179702783488 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _runtime‚4617.887970066 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407872 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177047040 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚81494918 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚50675413 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214665 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4472905369252 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚363311195321344 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4685924154880 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚321922401276468 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚367996862481920 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 180574076 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚363311171381248 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚317449495907216 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26088 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚324690060396032 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955591168 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚324690042393088 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚99079156 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1212 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411420 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 256994304 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚321921486825688 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚50675415 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚329179702783488 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _runtime‚4617.887970066 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407872 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177047040 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚81494918 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚50675413 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214665 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4472905369252 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚363311195321344 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4685924154880 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚321922401276468 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚367996862481920 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 180574076 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚363311171381248 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚317449495907216 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26088 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚324690060396032 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955591168 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚324690042393088 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚99079156 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1212 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411420 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 256994304 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚321921486825688 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚50675415 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚329178785372672 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4489642387456 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚329178785372672 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26088 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚329179702783488 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 233054208 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955591168 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _step‚322 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26090 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚591 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1187 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410816 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407872 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚74401593 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚81494918ÂS7§æ2¤- _wandbruntime‚4619  _runtime‚4619Ö9N ¼¥öM WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955591168 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917424128 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 222555136 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4474014030778 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚50687789 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 180618591 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899421184 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚317527476001344 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007241 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚81514818 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4687078006272 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 trainer/global_step‚3949 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 180618591 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚324769672688640 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4490755204096 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4489855782912 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚81514816 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚74419546 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 180592475 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚329260445895680 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚322001490032122 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4473116619150 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚99077659 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚324769672688640 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4490755204096 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4489855782912 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 _step‚323 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚99103773 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4686855451136 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 % _timestamp‚1.7885149937141206e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚317527458961984 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26114 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 _runtime‚4619.185347766 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚324769690691584 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚23732356 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚599 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚23731757 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚74420147 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚324769690691584 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177047040 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚329259528471552 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1187 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚81514818 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 246495232 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚363400490601984 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899421184 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31686 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 123921070żi3@08 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚99103773 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚329260445895680 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26114 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚368087592548352 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚81514816 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 180592475 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚99077659 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚74420147 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚324769690691584 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177047040 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚329259528471552 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1187 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚81514818 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 246495232 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚363400490601984 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899421184 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31686 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚99103773 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚329260445895680 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26114 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚368087592548352 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚81514816 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 180592475 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚99077659 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955591168 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917424128 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 222555136 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4474014030778 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚50687789 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 180618591 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899421184 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚317527476001344 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007241 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚81514818 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4687078006272 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 trainer/global_step‚3949 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 180618591 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚324769672688640 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4490755204096 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4489855782912 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚81514816 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚74419546 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 180592475 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚329260445895680 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416Ã'@>¬2§- _wandbruntime‚4619  _runtime‚4619*+dAW¨Ñ 0loss/FRt4/assignment_loss‚2.5531373284757137 0loss/FBt2/assignment_loss‚0.0515641589545055 .loss/FBt3/detection_loss‚2.092853014667829 /loss/FRt3/assignment_loss‚3.010525654662739 /loss/FRt2/detection_loss‚1.9952989276912478 /loss/FRt1/assignment_loss‚3.192858507438582 1loss/SRqqt3/assignment_loss‚1.235345787107291 0loss/FBt4/assignment_loss‚0.3691172320728323 /loss/FRt4/detection_loss‚2.7003531232476234 1loss/SRqqt1/detection_loss‚1.6997322829750676 /loss/FBt2/detection_loss‚0.5081147204211447 .loss/FBt4/detection_loss‚2.751406865460532 %loss/total_loss‚40.60327469275727 _runtime‚4619.187059164 0loss/FRt2/assignment_loss‚1.5312167776955499 1loss/FBt3/assignment_loss‚0.20664160312298918 /loss/FRt1/detection_loss‚2.3304634215880413 trainer/global_step‚3949 % _timestamp‚1.7885149937169447e+09 1loss/SRqqt1/assignment_loss‚1.604854759660763 0loss/FBt1/assignment_loss‚0.1626416444113212 /loss/FRt3/detection_loss‚2.3219583943824875 1loss/SRqqt2/detection_loss‚1.0975033204174705 0loss/SRqqt4/detection_loss‚2.086264518234465 1loss/SRqqt4/assignment_loss‚1.843333340353436 1loss/SRqqt3/detection_loss‚1.8217254364699649 /loss/FBt1/detection_loss‚2.5610068568161557 _step‚324 1loss/SRqqt2/assignment_loss‚0.875357016429512 epoch‚3;äÂL\©Ö /loss/FRt3/assignment_loss‚3.010525654662739 /loss/FRt2/detection_loss‚1.9952989276912478 /loss/FRt1/assignment_loss‚3.192858507438582 1loss/SRqqt3/assignment_loss‚1.235345787107291 0loss/FBt4/assignment_loss‚0.3691172320728323 /loss/FRt4/detection_loss‚2.7003531232476234 1loss/SRqqt1/detection_loss‚1.6997322829750676 /loss/FBt2/detection_loss‚0.5081147204211447 .loss/FBt4/detection_loss‚2.751406865460532 %loss/total_loss‚40.60327469275727 _runtime‚4619.187059164 0loss/FRt2/assignment_loss‚1.5312167776955499 1loss/FBt3/assignment_loss‚0.20664160312298918 /loss/FRt1/detection_loss‚2.3304634215880413 trainer/global_step‚3949 % _timestamp‚1.7885149937169447e+09 1loss/SRqqt1/assignment_loss‚1.604854759660763 0loss/FBt1/assignment_loss‚0.1626416444113212 /loss/FRt3/detection_loss‚2.3219583943824875 1loss/SRqqt2/detection_loss‚1.0975033204174705 0loss/SRqqt4/detection_loss‚2.086264518234465 1loss/SRqqt4/assignment_loss‚1.843333340353436 1loss/SRqqt3/detection_loss‚1.8217254364699649 /loss/FBt1/detection_loss‚2.5610068568161557 _step‚324 1loss/SRqqt2/assignment_loss‚0.875357016429512 epoch‚3 0loss/FRt4/assignment_loss‚2.5531373284757137 0loss/FBt2/assignment_loss‚0.0515641589545055 .loss/FBt3/detection_loss‚2.092853014667829Ä=ã/û”ª‚(:‰ ²¥êÔãÿ gpu.0.gpu‚28.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚170.559' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.63975 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‹‰×Rd«‚(:Ù ²¥êÔ€»Ì€ network.sent‚ 9.44157e+06) disk./.usagePercent‚75.85165117756641 disk.sda2.in‚0.09375! disk.sda2.out‚531.40087890625 disk.md0.in‚ 0.17578125 proc.cpu.threads‚17 disk.rbd0.in‚0! disk.nvme1n1.out‚ 29407.828125 disk.rbd2.in‚0 disk.md0p1.out‚ 512.04296875 network.recv‚ 1.530745e+06 disk.sdb1.in‚0 disk.sda.in‚0.09375$ memory_percent‚96.01292610168457# proc.memory.rssMB‚ 2990.49609375 disk.rbd0.out‚1.125 disk.sda1.in‚0 disk.nvme0n1.in‚ 603.27734375 disk.sdb.out‚531.40087890625 disk.nvme1n1.in‚ 498.61328125 disk.dm-0.in‚ 1101.890625) proc.memory.percent‚18.25253963470459 disk.sdb1.out‚0 disk.dm-0.out‚53761.03515625 disk.md0p1.in‚ 0.17578125 disk.sdb.in‚ 0.08203125 cpu‚25.210218426608126$ disk./.usageGB‚2712.822452545166 disk.md0.out‚ 512.04296875 disk.rbd2.out‚1.0625 disk.sdb2.in‚ 0.08203125 disk.sda1.out‚0# disk.nvme0n1.out‚24353.20703125! disk.sdb2.out‚531.40087890625 disk.sda.out‚531.40087890625' proc.memory.availableMB‚ 653.2421875;8씬‚(:‰ Á¥êÔ¹òâÿ gpu.0.gpu‚45.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚163.597' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚40.89925 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚09óµc­‚(:Ø Á¥êÔ‡¦ûÿ disk.nvme0n1.in‚ 603.27734375 disk.sdb2.in‚ 0.08203125 disk.sdb.in‚ 0.08203125 disk.md0.in‚ 0.17578125 network.sent‚ 9.537673e+06 disk.dm-0.in‚ 1101.890625" disk.nvme0n1.out‚ 24402.5234375) disk./.usagePercent‚75.85025051145858 disk.dm-0.out‚54110.50390625 disk.md0p1.in‚ 0.17578125 disk.sda.in‚0.09375 disk.sda.out‚533.28662109375 disk.nvme1n1.in‚ 498.61328125 disk.rbd2.out‚1.0625 disk.sdb1.out‚0 disk.rbd2.in‚0# proc.memory.rssMB‚ 2990.49609375 cpu‚25.20949538998397 proc.cpu.threads‚17! disk.sdb2.out‚533.28662109375 disk.sda1.out‚0 disk.md0p1.out‚ 513.8671875 disk.rbd0.in‚0 disk.rbd0.out‚1.125) proc.memory.percent‚18.25253963470459 network.recv‚ 1.535141e+06$ disk./.usageGB‚2712.772357940674 disk.sda1.in‚0 disk.sdb.out‚533.28662109375 disk.sdb1.in‚0$ memory_percent‚96.01168632507324 disk.sda2.in‚0.09375! disk.sda2.out‚533.28662109375# disk.nvme1n1.out‚29707.98046875 disk.md0.out‚ 513.8671875' proc.memory.availableMB‚ 653.4453125/?·ç”®‚(:‰ Ð¥êÔÔοÿ gpu.0.gpu‚47.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚172.395' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.09875 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¡¼XFe¯‚(:Ú Ð¥êÔæÉÙÿ disk.sda.in‚0.09375 disk.sda2.in‚0.09375 disk.rbd2.in‚0 disk.dm-0.out‚ 54188.8828125 network.sent‚ 9.541406e+06 disk.md0.out‚ 515.48046875 disk.sdb2.out‚ 534.978515625 disk.dm-0.in‚ 1101.890625 disk.md0p1.in‚ 0.17578125" disk.nvme0n1.out‚ 24402.9609375 cpu‚25.208879760228676 proc.cpu.threads‚17 disk.sdb.in‚ 0.08203125 disk.sdb.out‚ 534.978515625 disk.sda.out‚ 534.978515625 disk.rbd0.in‚0 disk.rbd0.out‚ 1.13671875 disk.sdb1.in‚0 disk.rbd2.out‚ 1.13671875 disk.sda1.in‚0# proc.memory.rssMB‚ 2990.49609375 network.recv‚ 1.539553e+06 disk.sda2.out‚ 534.978515625 disk.md0.in‚ 0.17578125 disk.md0p1.out‚ 515.48046875 disk.sda1.out‚0# memory_percent‚96.0092306137085) disk./.usagePercent‚75.85235076399658 disk.nvme1n1.in‚ 498.61328125 disk.sdb2.in‚ 0.08203125( proc.memory.availableMB‚ 653.84765625 disk.nvme0n1.in‚ 603.27734375% disk./.usageGB‚2712.8474731445312! disk.nvme1n1.out‚ 29785.921875 disk.sdb1.out‚0) proc.memory.percent‚18.25253963470459àÁ±ª”°‚(:‰ ߥêÔöºÖÿ gpu.0.gpu‚16.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚162.481' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚40.62025 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ›ß^±‚(:Ó ߥêÔíë‹€ network.sent‚ 9.5495e+06 disk.sda1.in‚0 disk.sda1.out‚0 disk.md0.in‚ 0.17578125) proc.memory.percent‚18.25253963470459 disk.sdb2.in‚ 0.08203125 disk.sdb.in‚ 0.08203125 disk.sda2.out‚ 536.615234375 disk.nvme1n1.in‚ 498.61328125 cpu‚25.20831385721636 proc.cpu.threads‚17% disk./.usageGB‚2712.7745361328125 disk.rbd2.out‚ 1.13671875 disk.dm-0.out‚ 54470.296875 disk.rbd0.in‚0) disk./.usagePercent‚75.85031141462242 disk.sdb2.out‚ 536.615234375 disk.nvme0n1.in‚ 603.27734375" disk.nvme0n1.out‚ 24402.9765625 disk.sda.in‚0.09375 disk.sdb1.out‚0( proc.memory.availableMB‚ 653.23046875 disk.sdb1.in‚0 disk.md0.out‚ 517.046875# memory_percent‚96.0129976272583 disk.dm-0.in‚ 1101.890625 disk.md0p1.in‚ 0.17578125 disk.md0p1.out‚ 517.046875# proc.memory.rssMB‚ 2990.49609375 network.recv‚ 1.543416e+06 disk.rbd2.in‚0 disk.sda.out‚ 536.615234375 disk.rbd0.out‚ 1.19921875" disk.nvme1n1.out‚ 30067.3203125 disk.sdb.out‚ 536.615234375 disk.sda2.in‚0.09375¦îü2²- _wandbruntime‚4673  _runtime‚4673AÓKldzÁ trainer/global_step‚3999 % _timestamp‚1.7885150484049184e+09 _runtime‚4673.875270189 _step‚325 & lr-AdamW/pg1‚0.0007259253471630648 & lr-AdamW/pg2‚0.0007259253471630648ÜèÿÑÌ´Æ & lr-AdamW/pg2‚0.0007259253471630648 trainer/global_step‚3999 % _timestamp‚1.7885150484049184e+09 _runtime‚4673.875270189 _step‚325 & lr-AdamW/pg1‚0.0007259253471630648Å]/çæ2µ- _wandbruntime‚4673  _runtime‚4673Xk´ö'¶ðO GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450780 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚321348483574256 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 100285970 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4527454614940 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚333215954730496 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4545296967680 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917410816 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407872 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚333216872141312 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚75307518 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚372521313534464 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚333215954730496 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚372521067025920 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4744157176320 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚321348500613616 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1212 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 222568448 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚367777156358144 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955591168 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 246508544 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚75306902 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177047040 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚367777132418048 trainer/global_step‚3999 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 100285970 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955591168 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚24012689 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1187 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411420 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚82489818 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26090 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0ƒy;OŽ·õO H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1212 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 222568448 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚367777156358144 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955591168 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607Ïx.Çm# VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 246508544 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚75306902 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177047040 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚367777132418048 trainer/global_step‚3999 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 100285970 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955591168 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚24012689 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1187 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411420 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚82489818 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26090 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450780 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚321348483574256 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 100285970 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4527454614940 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚333215954730496 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4545296967680 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917410816 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407872 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚333216872141312 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚75307518 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚372521313534464 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚333215954730496 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚372521067025920 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4744157176320 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚321348500613616Æïè‹ç2¸- _wandbruntime‚4674  _runtime‚4674‡aºÖ'¹úM K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955591168 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _runtime‚4674.815785808 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚82509718 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚75325574 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚367866451638784 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚328751205469184 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1187 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚328751187466240 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26114 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4529460687886 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 % _timestamp‚1.7885150493448834e+09 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899421184 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚367866475578880 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4745313968640 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚82509716 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚372611532566528 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚325955941395630 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177047040 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 100310561 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4545510363136 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 182846393 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899421184 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚328751205469184 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚24018983 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 182820277 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 100336675 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚333296697829376 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450988 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚333297615253504 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚75324956 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31686 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚51306589 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚82509716 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 256980992®ç†'ºÿM >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚325955941395630 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177047040 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 100310561 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4545510363136 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 182846393 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899421184 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚328751205469184 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚24018983 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 182820277 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 100336675 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚333296697829376 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450988 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚333297615253504 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚75324956 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31686 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚51306589 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚82509716 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 256980992 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955591168 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _runtime‚4674.815785808 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚82509718 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚75325574 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚367866451638784 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚328751205469184 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1187 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚328751187466240 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26114 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4529460687886 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 % _timestamp‚1.7885150493448834e+09 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899421184 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚367866475578880 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4745313968640 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚82509716 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚372611532566528 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096Ç›Ÿ­2»- _wandbruntime‚4674  _runtime‚4674"ª°ôZ¼Ô 1loss/FBt2/assignment_loss‚0.13390440757963795 1loss/SRqqt1/detection_loss‚1.6092577103910775 0loss/FRt3/assignment_loss‚2.2055853969520993 /loss/FRt2/detection_loss‚1.6289138749072498 1loss/SRqqt3/detection_loss‚1.5809215104414358 % _timestamp‚1.7885150493471675e+09 2loss/SRqqt1/assignment_loss‚1.5052502840950057 .loss/FBt1/detection_loss‚2.482781672477722 /loss/FBt2/detection_loss‚0.7736505868772118 /loss/FBt3/detection_loss‚1.8947266382830483 epoch‚3 0loss/FRt2/assignment_loss‚1.4787686670368367 0loss/FRt4/assignment_loss‚3.7150247538531267 0loss/SRqqt2/assignment_loss‚1.05112551906777 0loss/FBt1/assignment_loss‚0.2513958802038644 /loss/FRt1/detection_loss‚2.2143690191367864 .loss/FRt3/detection_loss‚2.203540691760955 .loss/FRt4/detection_loss‚3.169634649047145 1loss/SRqqt4/detection_loss‚3.2757854869491174 trainer/global_step‚3999 _runtime‚4674.817287085 _step‚328 2loss/SRqqt3/assignment_loss‚1.1727697312555931 2loss/SRqqt4/assignment_loss‚2.0670762832619642 0loss/FBt4/assignment_loss‚0.4232073279087093 /loss/FBt4/detection_loss‚2.1791804649613122 /loss/FRt1/assignment_loss‚2.498084399294346 0loss/FBt3/assignment_loss‚0.3581194833636151 1loss/SRqqt2/detection_loss‚1.1488847118471899 %loss/total_loss‚41.02195915095282SÑ Ã_½Ù 0loss/FRt2/assignment_loss‚1.4787686670368367 0loss/FRt4/assignment_loss‚3.7150247538531267 0loss/SRqqt2/assignment_loss‚1.05112551906777 0loss/FBt1/assignment_loss‚0.2513958802038644 /loss/FRt1/detection_loss‚2.2143690191367864 .loss/FRt3/detection_loss‚2.203540691760955 .loss/FRt4/detection_loss‚3.169634649047145 1loss/SRqqt4/detection_loss‚3.2757854869491174 trainer/global_step‚3999 _runtime‚4674.817287085 _step‚328 2loss/SRqqt3/assignment_loss‚1.1727697312555931 2loss/SRqqt4/assignment_loss‚2.0670762832619642 0loss/FBt4/assignment_loss‚0.4232073279087093 /loss/FBt4/detection_loss‚2.1791804649613122 /loss/FRt1/assignment_loss‚2.498084399294346 0loss/FBt3/assignment_loss‚0.3581194833636151 1loss/SRqqt2/detection_loss‚1.1488847118471899 %loss/total_loss‚41.02195915095282 1loss/FBt2/assignment_loss‚0.13390440757963795 1loss/SRqqt1/detection_loss‚1.6092577103910775 0loss/FRt3/assignment_loss‚2.2055853969520993 /loss/FRt2/detection_loss‚1.6289138749072498 1loss/SRqqt3/detection_loss‚1.5809215104414358 % _timestamp‚1.7885150493471675e+09 2loss/SRqqt1/assignment_loss‚1.5052502840950057 .loss/FBt1/detection_loss‚2.482781672477722 /loss/FBt2/detection_loss‚0.7736505868772118 /loss/FBt3/detection_loss‚1.8947266382830483 epoch‚3È•žÓ”¾‚(:‰ î¥êÔàî…€ gpu.0.gpu‚55.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚173.169' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.29225 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0bß0!ž¿‚(:á î¥êÔƶ¹€ disk.sdb.in‚ 0.08203125 disk.nvme1n1.in‚ 498.61328125 disk.sdb2.in‚ 0.08203125 proc.cpu.threads‚17) disk./.usagePercent‚75.84389706354169 disk.dm-0.in‚ 1101.890625 disk.md0p1.in‚ 0.17578125 disk.sda.in‚0.09375 disk.rbd0.in‚0 disk.nvme0n1.in‚ 603.27734375 disk.sda2.in‚0.09375 disk.rbd0.out‚ 1.19921875 disk.md0.in‚ 0.17578125 dis–/ò¾Îk.md0.out‚ 518.9765625 cpu‚25.207645797575047 network.sent‚ 9.553749e+06 network.recv‚ 1.549e+06 disk.sda1.in‚0 disk.md0p1.out‚ 518.9765625! disk.sda2.out‚538.62060546875! disk.sdb2.out‚538.62060546875# proc.memory.rssMB‚ 2990.49609375 disk.sda1.out‚0 disk.sda.out‚538.62060546875 disk.sdb1.in‚0 disk.sdb1.out‚0$ memory_percent‚96.01359367370605) proc.memory.percent‚18.25253963470459 disk.rbd2.out‚ 1.2158203125' proc.memory.availableMB‚ 653.1328125% disk./.usageGB‚2712.5451278686523# disk.nvme0n1.out‚24403.18359375 disk.sdb.out‚538.62060546875 disk.dm-0.out‚54528.44921875! disk.nvme1n1.out‚ 30125.265625 disk.rbd2.in‚0ýæ”À‚(:‰ ý¥êÔ⪠ÿ gpu.0.gpu‚54.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚170.559' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.63975 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0xá9õoÁ‚(:ä ý¥êÔï—Æÿ# memory_percent‚96.0123062133789% disk./.usageGB‚2712.7942390441895 disk.sda2.in‚0.09375 disk.sdb1.out‚0 network.sent‚ 9.660476e+06 network.recv‚ 1.553163e+06 disk.nvme1n1.in‚ 498.61328125) proc.memory.percent‚18.25253963470459 disk.sdb1.in‚0! disk.sdb2.out‚540.17431640625 disk.dm-0.out‚54685.00390625 disk.md0p1.out‚ 520.4609375 disk.sda.in‚0.09375 cpu‚25.207048993296574 proc.cpu.threads‚17 disk.rbd0.in‚0 disk.rbd2.out‚ 1.2158203125 disk.sdb.out‚540.17431640625 disk.sdb2.in‚ 0.08203125 disk.sda1.out‚0 disk.md0p1.in‚ 0.17578125" disk.nvme0n1.out‚ 24403.3359375 disk.sdb.in‚ 0.08203125# disk.nvme1n1.out‚30281.66796875 disk.md0.in‚ 0.17578125 disk.nvme0n1.in‚ 603.27734375 disk.sda.out‚540.17431640625% proc.memory.availableMB‚ 653.34375! disk.sda2.out‚540.17431640625 disk.rbd0.out‚ 1.2783203125 disk.dm-0.in‚ 1101.890625# proc.memory.rssMB‚ 2990.49609375) disk./.usagePercent‚75.85086231627083 disk.md0.out‚ 520.4609375 disk.rbd2.in‚0 disk.sda1.in‚0‹ =œ^‚(:Ó Œ¦êÔ“ßþ' proc.memory.availableMB‚ 653.1015625 proc.cpu.threads‚17 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.md0.out‚ 521.96484375 disk.sda.out‚ 541.734375 disk.sda2.in‚0.09375 disk.rbd2.out‚ 1.27880859375 disk.sdb2.in‚ 0.08203125 disk.md0p1.out‚ 521.96484375 disk.nvme0n1.in‚ 603.27734375 disk.sda2.out‚ 541.734375$ memory_percent‚96.01378440856934 disk.rbd0.out‚ 1.2783203125# proc.memory.rssMB‚ 2990.49609375# disk.nvme1n1.out‚30412.48046875 disk.rbd2.in‚0 disk.sdb2.out‚ 541.734375 network.recv‚ 1.557445e+06 disk.md0.in‚ 0.17578125# disk.nvme0n1.out‚24404.58203125 disk.nvme1n1.in‚ 498.61328125 disk.sda1.in‚0 disk.dm-0.out‚ 54817.0625 disk.sda.in‚0.09375 cpu‚25.206507724172514% disk./.usageGB‚2712.7076873779297 disk.rbd0.in‚0 disk.dm-0.in‚ 1101.890625) proc.memory.percent‚18.25253963470459 network.sent‚ 9.666522e+06) disk./.usagePercent‚75.84844229545752 disk.sda1.out‚0 disk.md0p1.in‚ 0.17578125 disk.sdb.in‚ 0.08203125 disk.sdb.out‚ 541.734375©NHהÂ(:‰ Œ¦êÔŠÚý gpu.0.gpu‚56.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚39.0 gpu.0.powerWatts‚179.397' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚44.84925 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0£ÎŠ“Ä‚(:ˆ ›¦êÔÈ€ðý gpu.0.gpu‚49.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚156.294' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚39.0735 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0|{grÅ‚(:ç ›¦êÔŒŒøþ network.sent‚ 9.672475e+06 disk.md0p1.in‚ 0.17578125# disk.nvme0n1.out‚24475.33203125 network.recv‚ 1.561334e+06 disk.dm-0.in‚ 1101.890625 disk.sda.in‚0.09375 disk.sdb1.in‚0 disk.sda1.out‚0) disk./.usagePercent‚75.85231385945248% disk./.usageGB‚2712.8461532592773 disk.sdb2.in‚ 0.08203125 proc.cpu.threads‚17 disk.dm-0.out‚55089.98828125 disk.md0p1.out‚ 523.38671875 disk.sdb.in‚ 0.08203125 disk.md0.in‚ 0.17578125 disk.rbd2.out‚ 1.27880859375) proc.memory.percent‚18.25253963470459 cpu‚25.205958791388444 disk.nvme0n1.in‚ 603.27734375 disk.sdb1.out‚0 disk.md0.out‚ 523.38671875 disk.sdb.out‚543.22998046875$ memory_percent‚96.01383209228516 disk.sda.out‚543.22998046875 disk.rbd0.in‚0 disk.nvme1n1.out‚ 30614.65625 disk.rbd2.in‚0! disk.sdb2.out‚543.22998046875 disk.sda1.in‚0% proc.memory.availableMB‚ 653.09375 disk.sda2.in‚0.09375! disk.sda2.out‚543.22998046875 disk.rbd0.out‚ 1.34130859375 disk.nvme1n1.in‚ 498.61328125# proc.memory.rssMB‚ 2990.49609375ßýl2Æ- _wandbruntime‚4726  _runtime‚4726¶æaÒÇÇÁ _step‚329 & lr-AdamW/pg1‚0.0007214296242134224 & lr-AdamW/pg2‚0.0007214296242134224 trainer/global_step‚4049 % _timestamp‚1.7885151008761377e+09 _runtime‚4726.346706579š¯ê-ÌÈÆ % _timestamp‚1.7885151008761377e+09 _runtime‚4726.346706579 _step‚329 & lr-AdamW/pg1‚0.0007214296242134224 & lr-AdamW/pg2‚0.0007214296242134224 trainer/global_step‚4049É:´p×2É- _wandbruntime‚4726  _runtime‚4726 *5ö'ÊðO ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚377045275788800 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚83484718 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚337253124088320 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410816 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4600052140032 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚83484720 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4600951547904 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4802182333952 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚76213034 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚329830389552704 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚638 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 242314240 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚4726.348204756 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407872 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 218374144 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚337253124088320 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4600951547904 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4582901272048 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26090 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4583798683468 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 % _timestamp‚1.7885151008775613e+09 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚332653071948288 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407872 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 218374144 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚337253124088320 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4600951547904 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4582901272048 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26090 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4583798683468 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 % _timestamp‚1.7885151008775613e+09 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚332653071948288 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚337253124088320 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410816 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4600052140032 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚83484720 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4600951547904 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4802182333952 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚76213034 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚329830389552704 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚638 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 242314240 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚4726.348204756 JDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 101543463 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 185048079 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚24305767 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚83504616 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚332732720246784 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚332732702243840 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚51925391 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450988 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 101569577 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 _runtime‚4727.408634096 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26114 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚332732702243840 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4803560859136 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899421184 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4584907344994 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4602064364544 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4601164943360 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 256980992 $ _timestamp‚1.788515101937353e+09 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899421184 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177047040 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚337333867187200 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411628 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4602064364544 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 trainer/global_step‚4049 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚76230522 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚51925389 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚337334784611328 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4601164943360 NADeviceStÎà ‹JatsMonitor.on_train_batch_end/allocation.large_pool.freed‚83504616 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚325325485414144 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚76231158 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚332732720246784 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007241 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 185074195 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚83504618 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1187 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚329909478308150 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 101569577 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚332732720246784 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚332732702243840 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚51925391 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450988 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 101569577 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 _runtime‚4727.408634096 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26114 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚332732702243840 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4803560859136 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899421184 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4584907344994 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4602064364544 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4601164943360 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 256980992 $ _timestamp‚1.788515101937353e+09 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899421184 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177047040 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚337333867187200 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411628 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4602064364544 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 trainer/global_step‚4049 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚76230522 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚51925389 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚337334784611328 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4601164943360 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚83504616 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚325325485414144 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚76231158 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚332732720246784 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007241 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 185074195 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚83504618 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1187 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚329909478308150 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 101569577 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 101543463 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 185048079 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚24305767 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚83504616 EDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚341290293446144 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917410816 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4656606128128 % _timestamp‚1.7885151547294443e+09 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4656606128128 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚336634604728832 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚376709078431744 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 187257482 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚84479618 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚24587156 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚84479618 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26090 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚77118368 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚601 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚84479620 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 226762752 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚341290293446144 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917410816 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4656606128128 % _timestamp‚1.7885151547294443e+09 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4656606128128 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚336634604728832 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚376709078431744 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 187257482 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚84479618 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚24587156 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚84479618 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26090 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚77118368 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚601 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚84479620 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 226762752 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚77136454 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚84499516 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955591168 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚52544189 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚376798373712384 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917424128 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 187301997 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 trainer/global_step‚4099 _step‚335 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚341371036545024 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 102802479 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411628 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177047040 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 102802479 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4656819523584 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚381660154083328 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚376798397652480 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚329224490120544 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4640354002102 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 102776365 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 % _timestamp‚1.7885151557642944e+09 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899421184 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4657718944768 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚341371953969152 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 102776365 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚626 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚341371953969152 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚333864844122646 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚336714217021440 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚628 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚24592891 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚336714235024384 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚84499518 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚381659905490944 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 224652288 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚77137082 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 187275881 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚336714217021440 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899421184 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4861531778560 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚52544191 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2lª¹?'àÿM NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚84499516 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 398533632 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007241 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411628 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177047040 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 102802479 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4656819523584 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚381660154083328 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚376798397652480 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚329224490120544 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4640354002102 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 102776365 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 % _timestamp‚1.7885151557642944e+09 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899421184 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4657718944768 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚341371953969152 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 102776365 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚626 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚341371953969152 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚333864844122646 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚336714217021440 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚628 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚24592891 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚336714235024384 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚84499518 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚381659905490944 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 224652288 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚77137082 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 187275881 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚336714217021440 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899421184 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4861531778560 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚52544191 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚84499518 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4657718944768 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31686 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚24592265 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚333863929671658 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚77136454 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚84499516 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955591168 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚52544189 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚376798373712384 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917424128 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 187301997 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 trainer/global_step‚4099 _step‚335 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚341371036545024 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 102802479 F Qr1.18408203125( proc.memory.availableMB‚ 653.66796875# disk.nvme0n1.out‚24878.38671875 network.sent‚ 9.796978e+06 disk.sdb1.in‚0! disk.sdb2.out‚551.18408203125 disk.md0.out‚ 531.0546875 disk.sdb2.in‚0.828125 disk.dm-0.out‚55586.48046875 disk.nvme1n1.out‚ 30708.09375 disk.sdb.out‚551.18408203125 proc.cpu.threads‚17! disk.sda2.out‚551.18408203125 disk.nvme1n1.in‚ 498.61328125 disk.sdb1.out‚0 network.recv‚ 1.579035e+06) disk./.usagePercent‚75.85238692191697 disk.sda2.in‚ 4.30078125 disk.rbd0.in‚0 disk.rbd0.out‚ 1.404296875 disk.rbd2.in‚0¢ÃM”å‚(:‰ צêÔø«Ø gpu.0.gpu‚31.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚37.0 gpu.0.powerWatts‚116.583' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚29.14575 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ò §T”æ‚(:‰ æ¦êÔЊ…þ gpu.0.gpu‚38.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚36.0 gpu.0.powerWatts‚117.193' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚29.29825 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0m”(Zjç‚(:ß æ¦êÔýûèÿ network.recv‚ 1.584794e+06 disk.sda2.out‚552.6787109375 disk.rbd0.out‚ 1.404296875 disk.sdb2.out‚552.6787109375 disk.dm-0.out‚55737.05859375 disk.md0p1.in‚ 5.12890625# memory_percent‚96.0134744644165# disk.nvme1n1.out‚30708.37890625 disk.sda1.out‚0 disk.md0.out‚ 532.484375 disk.rbd2.in‚0 disk.rbd2.out‚ 1.341796875( proc.memory.availableMB‚ 653.15234375 disk.nvme1n1.in‚ 498.61328125 disk.md0p1.out‚ 532.484375 proc.cpu.threads‚17) disk./.usagePercent‚75.85329950942982 disk.sdb.out‚552.6787109375 disk.sda.in‚ 4.30078125 disk.rbd0.in‚0 disk.md0.in‚ 5.12890625 disk.nvme0n1.in‚ 603.27734375 network.sent‚ 9.906989e+06) proc.memory.percent‚18.25253963470459 disk.sdb.in‚0.828125 disk.sda.out‚552.6787109375 disk.sdb1.in‚0 disk.dm-0.in‚ 1101.890625 cpu‚25.20311069141274$ disk./.usageGB‚2712.881404876709 disk.sda2.in‚ 4.30078125 disk.sdb1.out‚0 disk.sdb2.in‚0.828125 disk.sda1.in‚0" disk.nvme0n1.out‚ 25028.6796875# proc.memory.rssMB‚ 2990.49609375³DÚó“è‚(:ˆ õ¦êÔàÖý gpu.0.gpu‚2.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚36.0 gpu.0.powerWatts‚114.131' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.53275 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0XQT²cé‚(:Ø õ¦êÔÉ·µÿ disk.sda.out‚ 554.208984375 disk.rbd0.in‚0 disk.md0.in‚ 5.12890625 disk.sdb2.out‚ 554.208984375 disk.sda1.in‚0' proc.memory.availableMB‚ 653.1484375 proc.cpu.threads‚17 disk.dm-0.in‚ 1101.890625 disk.nvme0n1.in‚ 603.27734375 disk.sda2.out‚ 554.208984375 disk.rbd0.out‚ 1.404296875) proc.memory.percent‚18.25253963470459 network.sent‚ 9.910144e+06 disk.md0.out‚533.9375 disk.rbd2.in‚0 network.recv‚ 1.588141e+06 disk.dm-0.out‚55947.32421875 disk.nvme1n1.in‚ 498.61328125 disk.md0p1.out‚533.9375 disk.sda.in‚ 4.30078125 disk.sdb1.out‚0 cpu‚25.20258886841783# disk.nvme0n1.out‚25238.80078125 disk.rbd2.out‚ 1.341796875 disk.sdb2.in‚0.828125# proc.memory.rssMB‚ 2990.49609375% disk./.usageGB‚2713.0988388061523 disk.sdb.in‚0.828125 disk.sdb.out‚ 554.208984375 disk.sda2.in‚ 4.30078125 disk.sdb1.in‚0$ memory_percent‚96.01349830627441) disk./.usagePercent‚75.85937905310021" disk.nvme1n1.out‚ 30708.5234375 disk.sda1.out‚0 disk.md0p1.in‚ 5.12890625]+†»‘ê‚(:† „§êÔÊ™éÿ gpu.0.gpu‚5.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚162.204' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚40.551 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ºM¢„hë‚(:Ý „§êÔ¯£å€ disk.dm-0.out‚ 56359.671875 network.sent‚ 9.918313e+06 network.recv‚ 1.592225e+06 disk.sdb2.in‚0.828125! disk.sda2.out‚555.59521484375 disk.rbd2.in‚0$ memory_percent‚96.01202011108398 cpu‚25.201999610077728 disk.dm-0.in‚ 1101.890625 disk.md0p1.out‚ 535.2734375 disk.sda2.in‚ 4.30078125' disk./.usagePercent‚75.866376303989 disk.sda1.out‚0 disk.sda.out‚555.59521484375! disk.nvme1n1.out‚ 30708.828125 disk.nvme0n1.in‚ 603.27734375 disk.sdb.in‚0.828125 disk.sda.in‚ 4.30078125 disk.nvme1n1.in‚ 498.61328125 disk.rbd2.out‚ 1.341796875$ disk./.usageGB‚2713.349094390869 disk.rbd0.out‚ 1.404296875 disk.md0.in‚ 5.12890625# proc.memory.rssMB‚ 2990.49609375 disk.sdb.out‚555.59521484375 disk.sdb1.in‚0 disk.md0.out‚ 535.2734375) proc.memory.percent‚18.25253963470459 disk.md0p1.in‚ 5.12890625 disk.nvme0n1.out‚ 25661.59375! disk.sdb2.out‚555.59521484375 disk.sda1.in‚0& proc.memory.availableMB‚ 653.390625 disk.rbd0.in‚0 disk.sdb1.out‚0 proc.cpu.threads‚17ou²¡2ì- _wandbruntime‚4837  _runtime‚4837˜OÆíÀ _step‚337 & lr-AdamW/pg1‚0.0007122562086449763 & lr-AdamW/pg2‚0.0007122562086449763 trainer/global_step‚4149 % _timestamp‚1.7885152117994545e+09 _runtime‚ 4837.27009195?$ÆËîÅ & lr-AdamW/pg1‚0.0007122562086449763 & lr-AdamW/pg2‚0.0007122562086449763 trainer/global_step‚4149 % _timestamp‚1.7885152117994545e+09 _runtime‚ 4837.27009195 _step‚337Ñ+ë2ï- _wandbruntime‚4837  _runtime‚4837ܯsõ'ðïO P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214665 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4711361300480 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚24872100 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚381175039468544 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚85474520 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407872 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚78022713 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚53150613 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410816 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚85474518 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1187 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚345327462803968 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407872 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚340616119506432 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 189485284 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411420 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚85474518 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 % _timestamp‚1.7885152118013215e+09 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚345327462803968 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 104010764 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26088 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚386093597169664 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007241 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _runtime‚ 4837.27229917 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 189459194 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚53150615 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚386093847872512 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚333045497693456 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4712260708352 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚337739292279720 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450780 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚333045514732816 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4693794586264 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 226762752 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955591168 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚78023333 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚618 trainer/global_step‚4149 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26090 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚620 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚345328380214784 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177047040 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚340616101503488 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1212 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚24872718 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 398533632 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26088 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 103984676 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4711361300480 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚381175015528448 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4918808403968 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚337740206730500 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 250702848 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4694691997684 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728cZsðú'ñôO A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚345327462803968 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 104010764 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26088 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚386093597169664 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007241 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _runtime‚ 4837.27229917 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 189459194 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚53150615 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚386093847872512 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚333045497693456 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4712260708352 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚337739292279720 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450780 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚333045514732816 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4693794586264 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 226762752 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955591168 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚78023333 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚618 trainer/global_step‚4149 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26090 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚620 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚345328380214784 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177047040 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚340616101503488 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1212 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚24872718 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 398533632 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26088 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 103984676 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4711361300480 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚381175015528448 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4918808403968 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚337740206730500 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 250702848 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4694691997684 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214665 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4711361300480 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚24872100 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚381175039468544 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚85474520 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407872 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚78022713 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚53150613 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410816 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚85474518 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1187 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚345327462803968 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407872 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚340616119506432 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 189485284 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411420 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚85474518 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 % _timestamp‚1.7885152118013215e+09ÒTGøá2ò- _wandbruntime‚4838  _runtime‚4838Á*•‡óúM OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 104035381 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4713373524992 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4694903247582 % _timestamp‚1.7885152129050407e+09 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007241 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚53162989 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4712474103808 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚381264358689280 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚337819295486154 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚386184058329600 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4712474103808 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 trainer/global_step‚4149 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1187 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 104009267 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 104035381 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 _step‚339 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955591168 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚386184317407744 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚340695731799040 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 189529799 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚333123477787584 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411628 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 189503683 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177047040 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚85494416 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26114 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26114 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚345408205902848 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 189529799 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917424128 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚53162991 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚340695749801984 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 259078144 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 235138048 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177047040 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚85494418 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4695800659210 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚78040784 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚08@*3'ôÿM TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177047040 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚85494418 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4695800659210 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚78040784 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 104035381 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4713373524992 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4694903247582 % _timestamp‚1.7885152129050407e+09 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007241 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚53162989 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4712474103808 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚381264358689280 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚337819295486154 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚386184058329600 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4712474103808 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 trainer/global_step‚4149 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1187 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 104009267 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 104035381 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 _step‚339 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955591168 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚386184317407744 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚340695731799040 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 189529799 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚333123477787584 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411628 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 189503683 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177047040 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚85494416 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26114 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26114 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚345408205902848 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 189529799 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917424128 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚53162991 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚340695749801984 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 259078144 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 235138048 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712Ó¯†’û2õ- _wandbruntime‚4838  _runtime‚4838ïuGaöÛ 0loss/FRt2/assignment_loss‚1.1444153857642207 1loss/SRqqt4/assignment_loss‚1.434243755923076 1loss/SRqqt2/detection_loss‚1.0043296262000998 2loss/SRqqt3/assignment_loss‚1.5233272208521764 /loss/FRt4/detection_loss‚3.1440918135643003 /loss/FRt1/detection_loss‚1.9879475539603133 trainer/global_step‚4149 0loss/FRt3/assignment_loss‚2.6687509490222467 /loss/FRt3/detection_loss‚2.4829862742889217 /loss/FBt1/detection_loss‚2.8577192028363547 /loss/FBt4/detection_loss‚1.9654865089584799 2loss/SRqqt1/assignment_loss‚1.2465405043553222 1loss/FBt1/assignment_loss‚0.17958248638509153 /loss/FRt2/detection_loss‚1.5425159959443684 1loss/SRqqt3/detection_loss‚1.7630525254644454 /loss/FBt2/detection_loss‚0.6337348327962393 _runtime‚4838.377606333 _step‚340 epoch‚3 % _timestamp‚1.7885152129074783e+09 0loss/FRt1/assignment_loss‚2.4100405649705365 1loss/FBt3/assignment_loss‚0.34499721306633546 0loss/FBt4/assignment_loss‚0.3636981833397466 1loss/SRqqt1/detection_loss‚1.7386884522031654 1loss/SRqqt4/detection_loss‚2.4796119621305754 .loss/FBt3/detection_loss‚2.933618262410164 0loss/FRt4/assignment_loss‚3.3974374389648436 1loss/FBt2/assignment_loss‚0.13869220697315962 2loss/SRqqt2/assignment_loss‚1.0728510796713333 &loss/total_loss‚40.458360000045516ö&åf÷à /loss/FRt1/detection_loss‚1.9879475539603133 trainer/global_step‚4149 0loss/FRt3/assignment_loss‚2.6687509490222467 /loss/FRt3/detection_loss‚2.4829862742889217 /loss/FBt1/detection_loss‚2.8577192028363547 /loss/FBt4/detection_loss‚1.9654865089584799 2loss/SRqqt1/assignment_loss‚1.2465405043553222 1loss/FBt1/assignment_loss‚0.17958248638509153 /loss/FRt2/detection_loss‚1.5425159959443684 1loss/SRqqt3/detection_loss‚1.7630525254644454 /loss/FBt2/detection_loss‚0.6337348327962393 _runtime‚4838.377606333 _step‚340 epoch‚3 % _timestamp‚1.7885152129074783e+09 0loss/FRt1/assignment_loss‚2.4100405649705365 1loss/FBt3/assignment_loss‚0.34499721306633546 0loss/FBt4/assignment_loss‚0.3636981833397466 1loss/SRqqt1/detection_loss‚1.7386884522031654 1loss/SRqqt4/detection_loss‚2.4796119621305754 .loss/FBt3/detection_loss‚2.933618262410164 0loss/FRt4/assignment_loss‚3.3974374389648436 1loss/FBt2/assignment_loss‚0.13869220697315962 2loss/SRqqt2/assignment_loss‚1.0728510796713333 &loss/total_loss‚40.458360000045516 0loss/FRt2/assignment_loss‚1.1444153857642207 1loss/SRqqt4/assignment_loss‚1.434243755923076 1loss/SRqqt2/detection_loss‚1.0043296262000998 2loss/SRqqt3/assignment_loss‚1.5233272208521764 /loss/FRt4/detection_loss‚3.1440918135643003ÔÞý&ò“ø‚(:ˆ “§êÔø‘‡€ gpu.0.gpu‚55.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚169.438' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.3595 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ît±iù‚(:Þ “§êÔ”–¥ disk.md0p1.out‚ 537.2890625 disk.sda2.out‚557.6806640625 proc.cpu.threads‚17( disk./.usagePercent‚75.8663683044491 disk.nvme0n1.in‚ 603.27734375 disk.sdb.in‚0.828125 disk.sdb1.out‚0 disk.rbd2.out‚ 1.341796875$ memory_percent‚96.01423740386963 disk.dm-0.in‚ 1101.890625# disk.nvme0n1.out‚25683.30859375 disk.sda.out‚557.6806640625 disk.rbd0.in‚0 disk.md0.out‚ 537.2890625# proc.memory.rssMB‚ 2990.49609375 disk.sdb2.in‚0.828125 disk.sdb2.out‚557.6806640625 disk.sda2.in‚ 4.30078125 disk.rbd0.out‚ 1.404296875) proc.memory.percent‚18.25253963470459 disk.sda1.in‚0 disk.sda.in‚ 4.30078125 disk.nvme1n1.in‚ 498.61328125 disk.md0.in‚ 5.12890625( proc.memory.availableMB‚ 653.02734375 cpu‚25.20142328540046 network.sent‚ 9.934865e+06 network.recv‚ 1.596167e+06$ disk./.usageGB‚2713.348808288574 disk.sda1.out‚0 disk.dm-0.out‚56392.24609375 disk.md0p1.in‚ 5.12890625 disk.sdb.out‚557.6806640625 disk.sdb1.in‚0 disk.nvme1n1.out‚ 30708.9375 disk.rbd2.in‚0o:™:’ú‚(:‡ ¢§êÔðÎßÿ gpu.0.gpu‚40.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚171.168' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.792 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¬%Žªgû‚(:Ü ¢§êÔ÷”ž€ disk.sdb2.in‚0.828125 disk.nvme0n1.in‚ 603.27734375 disk.rbd0.in‚0 disk.md0.in‚ 5.12890625 disk.rbd2.in‚0 network.recv‚ 1.6019e+06 disk.sda1.in‚0 disk.sdb1.in‚0$ memory_percent‚96.01430892944336 disk.sdb.in‚0.828125 disk.sda.in‚ 4.30078125# disk.nvme1n1.out‚30709.44140625 disk.sdb1.out‚0 disk.md0.out‚ 538.59375) disk./.usagePercent‚75.84850981157436$ disk./.usageGB‚2712.710102081299 disk.dm-0.in‚ 1101.890625 disk.md0p1.in‚ 5.12890625 disk.sda2.in‚ 4.30078125 disk.nvme1n1.in‚ 498.61328125 disk.rbd2.out‚ 1.341796875 cpu‚25.200765088698592! disk.sdb2.out‚559.03662109375 disk.sdb.out‚559.03662109375# proc.memory.rssMB‚ 2990.49609375) proc.memory.percent‚18.25253963470459 proc.cpu.threads‚17 disk.sda.out‚559.03662109375 disk.rbd0.out‚ 1.404296875& proc.memory.availableMB‚ 653.015625 network.sent‚ 1.003212e+07 disk.sda1.out‚0 disk.dm-0.out‚56784.00390625 disk.md0p1.out‚ 538.59375! disk.sda2.out‚559.03662109375 disk.nvme0n1.out‚ 26074.5625-­U´”ü‚(:‰ ±§êÔ›–¨ÿ gpu.0.gpu‚36.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚37.0 gpu.0.powerWatts‚162.481' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚40.62025 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0L Ãoý‚(:ä ±§êÔÉ„Ýÿ disk.md0p1.in‚ 5.12890625 disk.nvme0n1.in‚ 603.27734375 disk.sdb.out‚560.7724609375 disk.rbd0.in‚0 disk.rbd2.in‚0 network.recv‚ 1.605332e+06 disk.sda1.in‚0$ memory_percent‚96.01583480834961# proc.memory.rssMB‚ 2990.49609375 network.sent‚ 1.0037415e+07# disk.nvme1n1.out‚30709.56640625 disk.sdb1.out‚0 disk.sdb2.out‚560.7724609375 disk.dm-0.out‚56841.87109375 disk.sda2.out‚560.7724609375 cpu‚25.200105031533404 disk.sda1.out‚0" disk.nvme0n1.out‚ 26132.3046875 disk.rbd0.out‚ 1.404296875 disk.nvme1n1.in‚ 498.61328125 disk.md0.in‚ 5.12890625 disk.rbd2.out‚ 1.341796875 disk.sdb.in‚0.828125& proc.memory.availableMB‚ 652.765625) disk./.usagePercent‚75.85081442569191$ disk./.usageGB‚2712.792526245117 disk.dm-0.in‚ 1101.890625 disk.md0p1.out‚ 540.25390625 disk.sda.in‚ 4.30078125 disk.sda2.in‚ 4.30078125 disk.md0.out‚ 540.25390625 proc.cpu.threads‚17 disk.sdb2.in‚0.828125 disk.sda.out‚560.7724609375 disk.sdb1.in‚0) proc.memory.percent‚18.25253963470459Á¬wy”þ‚(:‰ À§êÔ²™¯ÿ gpu.0.gpu‚45.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚164.989' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚41.24725 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¥Ýn§eÿ‚(:Ú À§êÔ綨€ disk.md0.in‚ 5.12890625 disk.rbd0.out‚ 1.404296875' proc.memory.availableMB‚ 652.7421875 disk.rbd2.out‚ 1.341796875 disk.sdb2.in‚0.828125 disk.sdb2.out‚562.6279296875 disk.sda1.out‚0 disk.dm-0.out‚ 57006.4140625 disk.md0p1.in‚ 5.12890625 disk.sda2.in‚ 4.30078125 disk.nvme0n1.in‚ 603.27734375 disk.sdb.in‚0.828125# proc.memory.rssMB‚ 2990.49609375 disk.sdb1.in‚0 network.sent‚ 1.0041256e+07) disk./.usagePercent‚75.83605292802787" disk.nvme1n1.out‚ 30709.7890625 disk.sdb1.out‚0 disk.dm-0.in‚ 1101.890625 disk.sda2.out‚562.6279296875 disk.sdb.out‚562.6279296875 disk.sda.out‚562.6279296875 disk.md0.out‚ 542.03125 disk.rbd2.in‚0 disk.sda1.in‚0 disk.md0p1.out‚ 542.03125 disk.nvme0n1.out‚ 26296.625 disk.rbd0.in‚0$ memory_percent‚96.01597785949707 network.recv‚ 1.609537e+06 disk.sda.in‚ 4.30078125) proc.memory.percent‚18.25253963470459 cpu‚25.199538990624323 proc.cpu.threads‚17% disk./.usageGB‚2712.2645835876465 disk.nvme1n1.in‚ 498.61328125TŒÌ2€- _wandbruntime‚4890  _runtime‚4890-e–KÇÁ _runtime‚4890.540486136 _step‚341 & lr-AdamW/pg1‚0.0007075804277522278 & lr-AdamW/pg2‚0.0007075804277522278 trainer/global_step‚4199 % _timestamp‚1.7885152650700548e+09ýÛ{øÌ‚Æ _step‚341 & lr-AdamW/pg1‚0.0007075804277522278 & lr-AdamW/pg2‚0.0007075804277522278 trainer/global_step‚4199 % _timestamp‚1.7885152650700548e+09 _runtime‚4890.540486136Õk#†2ƒ- _wandbruntime‚4890  _runtime‚4890GÃŽ#$„ðO M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚78926786 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚25157982 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚385640976565248 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955591168 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407872 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177047040 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 191713086 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917410816 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 _runtime‚4890.542057864 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1212 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 398533632 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 220471296 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚349364632161792 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚25157373 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214665 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410816 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 105217578 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚86469418 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4767915288576 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚86469420 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 105217578 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚609 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4977026327552 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 _step‚342¥Â…û'…õO R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚349364632161792 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚25157373 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214665 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410816 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 105217578 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚86469418 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4767915288576 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚86469420 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 105217578 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚609 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4977026327552 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 _step‚342 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚78926786 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚25157982 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚385640976565248 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955591168 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407872 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177047040 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 191713086 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917410816 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 _runtime‚4890.542057864 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1212 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 398533632 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 220471296 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0Ö'±ÞK2†- _wandbruntime‚4891  _runtime‚4891J.¶'‡úM C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4769028105216 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚337022482493984 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177047040 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 191757601 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 398533632 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 105268283 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚25163738 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007241 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 % _timestamp‚1.7885152663251808e+09 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4750349904690 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 _runtime‚4891.797902391 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 191757601 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚341772832398674 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4768128684032 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917424128 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 191731485 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26114 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 233912320 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _step‚343 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚86489316 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚390708524126720 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4751247316318 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚344677246576640 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899421184 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 191731485 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚337022499533344 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚349445375260672 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177047040 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚611 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚86489318 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1187 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4978204400640 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚53781791 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚385730319726080 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450988 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955591168 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4768128684032 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚385730295785984 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚53781789 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 105242169 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚349446292684800 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚344677246576640 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899421184 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚86489316 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 105268283 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚78944918 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4977994428416 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564ê×÷'ˆÿM H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚609 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4750349904690 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 _runtime‚4891.797902391 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 191757601 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚341772832398674 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4768128684032 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917424128 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 191731485 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26114 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 233912320 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _step‚343 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚86489316 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚390708524126720 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4751247316318 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚344677246576640 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899421184 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 191731485 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚337022499533344 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚349445375260672 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177047040 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚611 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚86489318 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1187 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4978204400640 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚53781791 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚385730319726080 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450988 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955591168 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4768128684032 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚385730295785984 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚53781789 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 105242169 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚349446292684800 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚344677246576640 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899421184 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚86489316 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 105268283 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚78944918 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4977994428416 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4769028105216 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚337022482493984 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177047040 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 191757601 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 398533632 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 105268283 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚25163738 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007241 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 % _timestamp‚1.7885152663251808e+09 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 PDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚353401801519616 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 398533632 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5035006235136 trainer/global_step‚4249 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410816 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚353401801519616 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚390106961542144 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 106450480 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4823569868800 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407872 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚395141943837184 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚353402718930432 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚54388215 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4805585311900 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 106476568 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _step‚346 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚25444972 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4804687900480 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 218374144 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚340843507106256 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚348579149061632 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚348579131058688 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 106450480 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚395142186151424 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚79832572 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955591168 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4823569868800 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4822670460928 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚87464320 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚87464318 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407872 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955591168 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1187 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚390106937602048 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚353402718930432 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚25444359 _runtime‚4945.392998458 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 193940888 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚87464318 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚340843524145616 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411420 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 242314240 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 106476568 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1212 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚345649109457516 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450780 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 % _timestamp‚1.7885153199220793e+09 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214665 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 193914798 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚345648195006736 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26090 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26088 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚79833187 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984x@…€û'—õO ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚348579149061632 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚348579131058688 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 106450480 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚395142186151424 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚79832572 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955591168 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4823569868800 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4822670460928 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚87464320 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚87464318 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407872 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955591168 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1187 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚390106937602048 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚353402718930432 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚25444359 _runtime‚4945.392998458 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 193940888 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚87464318 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚340843524145616 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411420 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 242314240 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 106476568 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1212 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚345649109457516 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450780 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 % _timestamp‚1.7885153199220793e+09 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214665 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 193914798 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚345648195006736 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26090 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26088 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚79833187 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007241 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚348579131058688 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚348579149061632 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚615 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚613 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚353401801519616 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 398533632 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5035006235136 trainer/global_step‚4249 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410816 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚353401801519616 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚390106961542144 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 106450480 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4823569868800 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407872 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚395141943837184 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚353402718930432 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚54388215 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4805585311900 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 106476568 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _step‚346 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚25444972 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4804687900480 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 218374144 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚340843507106256 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171Úu#“2˜- _wandbruntime‚4946  _runtime‚4946$VP™ùM REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚54400589 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4823783264256 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 193985403 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 193985403 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5036170391040 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 106475071 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 224652288 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177047040 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚353483462042624 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚345728198213170 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 _runtime‚4946.419761843 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚87484218 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955591168 _step‚347 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4824682685440 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚87484218 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 398533632 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26114 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚25450609 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚25449996 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚54400591 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 $ _timestamp‚1.788515320948754e+09 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 193959287 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚348658779357184 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4806693973426 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 106501185 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177047040 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450988 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411628:&0¯ FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚87484216 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚348658779357184 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚79850585 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 248592384 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4805796561798 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 $ _timestamp‚1.788515320948754e+09 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 193959287 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚348658779357184 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4806693973426 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 106501185 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177047040 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450988 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411628 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚87484216 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚348658779357184 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚79850585 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 248592384 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4805796561798 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 193985403 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 193985403 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5036170391040 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 106475071 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 224652288 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177047040 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚353483462042624 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚345728198213170 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 _runtime‚4946.419761843 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚87484218 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955591168 _step‚347 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4824682685440 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚87484218 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 398533632 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26114 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚25450609 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚25449996 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚54400591 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665ÛT˜Ù2›- _wandbruntime‚4946  _runtime‚4946éÐ¥—YœÓ 1loss/FBt2/assignment_loss‚0.08111919475783794 /loss/FRt2/detection_loss‚1.4041814671122823 trainer/global_step‚4249 0loss/SRqqt3/detection_loss‚2.220746479251168 /loss/FBt2/detection_loss‚0.6897987489821389 0loss/FRt4/assignment_loss‚3.5493022902258513 2loss/SRqqt1/assignment_loss‚1.4858717111637816 2loss/SRqqt3/assignment_loss‚1.8083644749766046 /loss/FRt3/detection_loss‚2.7790152886882424 /loss/FBt1/detection_loss‚2.1337897817293805 .loss/FBt3/detection_loss‚2.667395282875408 /loss/FBt4/detection_loss‚2.6969609601157054 % _timestamp‚1.7885153209511833e+09 _step‚348 1loss/SRqqt1/detection_loss‚1.5120162962703034 1loss/SRqqt4/detection_loss‚2.7022011257134952 0loss/FBt1/assignment_loss‚0.1794516270545728 /loss/FRt2/assignment_loss‚1.377312051838842 2loss/SRqqt2/assignment_loss‚1.0829525997396559 2loss/SRqqt4/assignment_loss‚1.9274163393733592 -loss/FRt4/detection_loss‚3.00671426592202 0loss/FBt4/assignment_loss‚0.3448505967722407 /loss/FRt1/detection_loss‚2.1284641269185265 1loss/SRqqt2/detection_loss‚0.8818673637902571 epoch‚3 0loss/FBt3/assignment_loss‚0.2948612336238515 /loss/FRt1/assignment_loss‚2.722828163041009 /loss/FRt3/assignment_loss‚3.600338749587536 &loss/total_loss‚43.277820219524074 _runtime‚ 4946.42131199éFHâ^Ø 0loss/SRqqt3/detection_loss‚2.220746479251168 /loss/FBt2/detection_loss‚0.6897987489821389 0loss/FRt4/assignment_loss‚3.5493022902258513 2loss/SRqqt1/assignment_loss‚1.4858717111637816 2loss/SRqqt3/assignment_loss‚1.8083644749766046 /loss/FRt3/detection_loss‚2.7790152886882424 /loss/FBt1/detection_loss‚2.1337897817293805 .loss/FBt3/detection_loss‚2.667395282875408 /loss/FBt4/detection_loss‚2.6969609601157054 % _timestamp‚1.7885153209511833e+09 _step‚348 1loss/SRqqt1/detection_loss‚1.5120162962703034 1loss/SRqqt4/detection_loss‚2.7022011257134952 0loss/FBt1/assignment_loss‚0.1794516270545728 /loss/FRt2/assignment_loss‚1.377312051838842 2loss/SRqqt2/assignment_loss‚1.0829525997396559 2loss/SRqqt4/assignment_loss‚1.9274163393733592 -loss/FRt4/detection_loss‚3.00671426592202 0loss/FBt4/assignment_loss‚0.3448505967722407 /loss/FRt1/detection_loss‚2.1284641269185265 1loss/SRqqt2/detection_loss‚0.8818673637902571 epoch‚3 0loss/FBt3/assignment_loss‚0.2948612336238515 /loss/FRt1/assignment_loss‚2.722828163041009 /loss/FRt3/assignment_loss‚3.600338749587536 &loss/total_loss‚43.277820219524074 _runtime‚ 4946.42131199 1loss/FBt2/assignment_loss‚0.08111919475783794 /loss/FRt2/detection_loss‚1.4041814671122823 trainer/global_step‚4249ÜÇÚeÄ’ž‚(:‡ ü§êÔè°ÿ gpu.0.gpu‚40.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚163.09' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚40.7725 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0$UlYŸ‚(:Î ü§êÔ©£Õ network.sent‚ 1.0179183e+07 disk.sdb.in‚0.828125 disk.sda.in‚ 4.30078125 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.rbd2.out‚ 1.341796875 disk.sdb.out‚ 569.6015625 disk.sda2.out‚ 569.6015625 disk.sdb2.in‚0.828125 disk.sdb2.out‚ 569.6015625 proc.cpu.threads‚17 disk.sda.out‚ 569.6015625 disk.md0.in‚ 5.12890625 disk.md0p1.in‚ 5.12890625 disk.rbd0.in‚0 disk.md0.out‚548.75 disk.dm-0.out‚ 57470.7421875# memory_percent‚96.0158109664917( disk./.usagePercent‚75.8488934695084% disk./.usageGB‚2712.7238235473633 disk.nvme1n1.in‚ 498.61328125 disk.sda1.in‚0 disk.dm-0.in‚ 1102.05859375 disk.md0p1.out‚548.75# disk.nvme0n1.out‚26513.94140625# proc.memory.rssMB‚ 2990.49609375 disk.sda2.in‚ 4.30078125 disk.rbd0.out‚ 1.404296875# disk.nvme1n1.out‚30956.80078125 disk.rbd2.in‚0( proc.memory.availableMB‚ 652.76953125) proc.memory.percent‚18.25253963470459 cpu‚25.197419837624633 network.recv‚ 1.627978e+06 disk.sda1.out‚0 disk.nvme0n1.in‚ 603.4453125œæMÒ” ‚(:‰ ‹¨êÔœ¢’þ gpu.0.gpu‚54.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚166.207' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚41.55175 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 {Ñl¡‚(:á ‹¨êÔ˜ÛÄÿ network.sent‚ 1.0273683e+07 disk.dm-0.in‚ 1102.05859375" disk.nvme0n1.out‚ 26513.9765625( proc.memory.availableMB‚ 652.76953125# proc.memory.rssMB‚ 2990.49609375) proc.memory.percent‚18.25253963470459 cpu‚25.196949249423437% disk./.usageGB‚2713.1003761291504 disk.sdb1.out‚0 disk.md0.out‚ 550.2734375 disk.dm-0.out‚ 57470.9453125) disk./.usagePercent‚75.85942203729466 disk.rbd0.out‚ 1.404296875 disk.sdb.out‚571.1826171875 disk.sda2.in‚ 4.30078125 disk.md0.in‚ 5.12890625 disk.md0p1.in‚ 5.12890625 disk.nvme0n1.in‚ 603.4453125# memory_percent‚96.0158109664917 disk.sdb.in‚0.828125 disk.rbd2.out‚ 1.341796875 disk.sda1.out‚0 disk.sda.in‚ 4.30078125 disk.sda.out‚571.1826171875 disk.nvme1n1.out‚ 30956.96875 disk.sda1.in‚0 proc.cpu.threads‚17 network.recv‚ 1.633075e+06 disk.rbd0.in‚0 disk.nvme1n1.in‚ 498.61328125 disk.sdb1.in‚0 disk.sdb2.out‚571.1826171875 disk.sda2.out‚571.1826171875 disk.rbd2.in‚0 disk.sdb2.in‚0.828125 disk.md0p1.out‚ 550.2734375œ[mè’¢‚(:‡ š¨êÔó®ÿ gpu.0.gpu‚35.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚165.93' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚41.4825 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Z mo£‚(:ä š¨êÔ‘þÇ€ disk.md0.in‚ 5.12890625 disk.md0.out‚ 551.65234375 disk.dm-0.out‚58052.14453125 disk.md0p1.in‚ 5.12890625 disk.sda.out‚572.6162109375 disk.rbd0.out‚ 1.404296875 disk.nvme1n1.in‚ 498.61328125" disk.nvme1n1.out‚ 31307.1484375 disk.sdb1.out‚0 network.recv‚ 1.637746e+06 disk.sda1.out‚0 disk.md0p1.out‚ 551.65234375 disk.nvme0n1.in‚ 603.4453125 cpu‚25.19644802753413 disk.sda1.in‚0 disk.sda2.out‚572.6162109375$ memory_percent‚96.01726531982422 proc.cpu.threads‚17# disk.nvme0n1.out‚26744.99609375% proc.memory.availableMB‚ 652.53125# proc.memory.rssMB‚ 2990.49609375 network.sent‚ 1.0279939e+07 disk.dm-0.in‚ 1102.05859375 disk.sda2.in‚ 4.30078125 disk.rbd0.in‚0 disk.sda.in‚ 4.30078125) disk./.usagePercent‚75.86635475856151% disk./.usageGB‚2713.3483238220215 disk.sdb2.in‚0.828125 disk.sdb.in‚0.828125 disk.rbd2.in‚0 disk.sdb2.out‚572.6162109375 disk.sdb.out‚572.6162109375 disk.sdb1.in‚0 disk.rbd2.out‚ 1.341796875) proc.memory.percent‚18.25253963470459‚sãt“¤‚(:ˆ ©¨êÔõåÎÿ gpu.0.gpu‚3.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚36.0 gpu.0.powerWatts‚120.927' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚30.23175 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0XC§‘k¥‚(:à ©¨êÔÍÀƒÿ disk.rbd2.out‚ 1.341796875 disk.sdb.in‚0.828125 network.sent‚ 1.0286523e+07 disk.sdb2.in‚0.828125 disk.sda1.out‚0 disk.dm-0.in‚ 1102.05859375 disk.md0p1.in‚ 5.12890625 disk.rbd0.out‚ 1.404296875# disk.nvme1n1.out‚31371.35546875) proc.memory.percent‚18.25253963470459 disk.md0.in‚ 5.12890625 disk.md0.out‚ 553.9453125 disk.rbd0.in‚0 proc.cpu.threads‚17 disk.dm-0.out‚58188.44140625 disk.sda.in‚ 4.30078125 disk.nvme1n1.in‚ 498.61328125( proc.memory.availableMB‚ 653.00390625 disk.sdb2.out‚ 574.974609375 disk.sda1.in‚0 disk.md0p1.out‚ 553.9453125# proc.memory.rssMB‚ 2990.49609375 disk.sda2.out‚ 574.974609375 disk.sdb1.out‚0 network.recv‚ 1.640374e+06) disk./.usagePercent‚75.86636211813823 disk.rbd2.in‚0 disk.nvme0n1.in‚ 603.4453125" disk.nvme0n1.out‚ 26817.0859375 disk.sdb.out‚ 574.974609375 disk.sda.out‚ 574.974609375 disk.sda2.in‚ 4.30078125$ disk./.usageGB‚2713.348587036133 disk.sdb1.in‚0$ memory_percent‚96.01438045501709 cpu‚25.19597673920199äÏÅL2¦- _wandbruntime‚5000  _runtime‚5000Ì1 æç½ _step‚349 $ lr-AdamW/pg1‚0.00069805559167526 $ lr-AdamW/pg2‚0.00069805559167526 trainer/global_step‚4299 % _timestamp‚1.7885153750513175e+09 _runtime‚5000.521641044f"%aȨ $ lr-AdamW/pg1‚0.00069805559167526 $ lr-AdamW/pg2‚0.00069805559167526 trainer/global_step‚4299 % _timestamp‚1.7885153750513175e+09 _runtime‚5000.521641044 _step‚349ÝcH÷2©- _wandbruntime‚5000  _runtime‚5000xÈa #ªðO JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚394572898638848 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214665 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚352560663839232 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚357438970877440 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚399666423264256 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 107709470 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 107683382 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1212 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚55007015 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚349603560821024 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚344742511812656 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚399666180950016 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 398533632 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 _runtime‚5000.522449887 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4879224449024 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407872 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 218374144 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26090 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚352560663839232 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407872 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚88459218 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚25730163 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4860134557588 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007241 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5093500685312 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚349602646370244 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚344742528852016 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411420 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26088 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚352560645836288 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4879224449024 % _timestamp‚1.7885153750520008e+09 _step‚350 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚357439888288256 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450780 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177047040 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955591168 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 107683382 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 196142600 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4878325041152 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚25730786 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 trainer/global_step‚4299 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚88459220 WEDeviceSta¸Œ2$étsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4878325041152 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚55007013 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚394572922578944 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚357438970877440 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚357439888288256 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚352560645836288 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚80737176 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 107709470 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35÷@ Œû'«õO A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 107683382 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 196142600 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4878325041152 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚25730786 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 trainer/global_step‚4299 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚88459220 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4878325041152 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚55007013 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚394572922578944 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚357438970877440 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚357439888288256 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚352560645836288 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚80737176 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 107709470 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚394572898638848 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214665 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚352560663839232 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚357438970877440 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚399666423264256 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 107709470 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 107683382 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1212 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚55007015 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚349603560821024 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚344742511812656 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚399666180950016 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 398533632 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 _runtime‚5000.522449887 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4879224449024 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407872 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 218374144 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26090 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚352560663839232 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407872 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚88459218 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚25730163 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4860134557588 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007241 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5093500685312 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚349602646370244 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚344742528852016 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411420 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26088 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚352560645836288 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4879224449024 % _timestamp‚1.7885153750520008e+09 _step‚350 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚357439888288256 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450780 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177047040 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955591168 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0ÞMñƒ:2¬- _wandbruntime‚5001  _runtime‚5001CÈxß'­úM VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899421184 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26114 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚344820508946144 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚55019389 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007241 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚357519713976320 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚25736340 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 107734087 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚349682649576678 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 196213205 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚80755731 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚88479116 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 107707973 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚55019391 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚394662241799680 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚394662217859584 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5094663599104 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1187 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 107707973 _runtime‚5001.555938163 _step‚351 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚399756905398784 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5094434752512 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899421184 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚25735713 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚352640276131840 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚349681735125690 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177047040 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177047040 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 107734087 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31686 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 398533632 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26114 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚352640294134784 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚80755102 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 228846592 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 252786688 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411628 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 196213205 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4880337265664 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4879437844480 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4879437844480 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚357520631400448 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚399756652612096 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚352640276131840 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4861243218906 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728¡¯-'®ÿM XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚399756652612096 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚352640276131840 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4861243218906 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899421184 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26114 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚344820508946144 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚55019389 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007241 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚357519713976320 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚25736340 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 107734087 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚349682649576678 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 196213205 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚80755731 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚88479116 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 107707973 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚55019391 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚394662241799680 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚394662217859584 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5094663599104 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1187 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 107707973 _runtime‚5001.555938163 _step‚351 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚399756905398784 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5094434752512 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899421184 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚25735713 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚352640276131840 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚349681735125690 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177047040 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177047040 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 107734087 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31686 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 398533632 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26114 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚352640294134784 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚80755102 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 228846592 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 252786688 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411628 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 196213205 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4880337265664 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4879437844480 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4879437844480 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚357520631400448ß9†p2¯- _wandbruntime‚5001  _runtime‚5001uvc‚°Ô 1loss/SRqqt3/detection_loss‚1.8981442066951324 1loss/SRqqt4/detection_loss‚2.6048283309772096 -loss/FBt1/detection_loss‚2.18761382997036 % _timestamp‚1.7885153760878475e+09 _runtime‚5001.558221103 0loss/FBt3/assignment_loss‚0.8361402440017888 1loss/FBt4/assignment_loss‚0.17867888285157582 /loss/FBt2/detection_loss‚0.8117844242563782 0loss/FRt2/assignment_loss‚1.3490603372108105 2loss/SRqqt2/assignment_loss‚1.1580568499349315 0loss/FRt3/assignment_loss‚3.0286742374300957 1loss/FBt1/assignment_loss‚0.45799691014690325 1loss/FBt2/assignment_loss‚0.09850739672805019 epoch‚3 _step‚352 .loss/FRt1/assignment_loss‚2.89148917564979 0loss/FRt4/assignment_loss‚3.1897717351498813 /loss/FRt1/detection_loss‚2.5002728437001887 /loss/FBt4/detection_loss‚2.2286142806212106 trainer/global_step‚4299 1loss/SRqqt4/assignment_loss‚2.208094819609461 /loss/FRt2/detection_loss‚2.0764809055037277 1loss/SRqqt1/detection_loss‚1.8857014232625564 0loss/SRqqt2/detection_loss‚1.276407325319175 2loss/SRqqt3/assignment_loss‚1.5994858201277458 .loss/FRt4/detect÷ 4bØion_loss‚3.023406930591749 /loss/FBt3/detection_loss‚1.8890059803213393 2loss/SRqqt1/assignment_loss‚1.8106291892472655 /loss/FRt3/detection_loss‚2.2689146442959705 $loss/total_loss‚43.4577607236033ÓÆ…_±Ù -loss/FBt1/detection_loss‚2.18761382997036 % _timestamp‚1.7885153760878475e+09 _runtime‚5001.558221103 0loss/FBt3/assignment_loss‚0.8361402440017888 1loss/FBt4/assignment_loss‚0.17867888285157582 /loss/FBt2/detection_loss‚0.8117844242563782 0loss/FRt2/assignment_loss‚1.3490603372108105 2loss/SRqqt2/assignment_loss‚1.1580568499349315 0loss/FRt3/assignment_loss‚3.0286742374300957 1loss/FBt1/assignment_loss‚0.45799691014690325 1loss/FBt2/assignment_loss‚0.09850739672805019 epoch‚3 _step‚352 .loss/FRt1/assignment_loss‚2.89148917564979 0loss/FRt4/assignment_loss‚3.1897717351498813 /loss/FRt1/detection_loss‚2.5002728437001887 /loss/FBt4/detection_loss‚2.2286142806212106 trainer/global_step‚4299 1loss/SRqqt4/assignment_loss‚2.208094819609461 /loss/FRt2/detection_loss‚2.0764809055037277 1loss/SRqqt1/detection_loss‚1.8857014232625564 0loss/SRqqt2/detection_loss‚1.276407325319175 2loss/SRqqt3/assignment_loss‚1.5994858201277458 .loss/FRt4/detection_loss‚3.023406930591749 /loss/FBt3/detection_loss‚1.8890059803213393 2loss/SRqqt1/assignment_loss‚1.8106291892472655 /loss/FRt3/detection_loss‚2.2689146442959705 $loss/total_loss‚43.4577607236033 1loss/SRqqt3/detection_loss‚1.8981442066951324 1loss/SRqqt4/detection_loss‚2.6048283309772096à.?UQv²‚(:ë ¸¨êÔžŒŸÿ! disk.sdb2.out‚576.38818359375 disk.sda.out‚576.38818359375$ memory_percent‚96.01614475250244) proc.memory.percent‚18.25253963470459% disk./.usageGB‚2713.3485107421875! disk.sda2.out‚576.38818359375 disk.rbd0.out‚ 1.404296875 disk.sdb2.in‚0.828125 disk.nvme0n1.in‚ 603.4453125" disk.nvme0n1.out‚ 26860.7578125 network.recv‚ 1.644703e+06 disk.sdb1.out‚0 disk.md0.in‚ 5.12890625 disk.md0p1.out‚ 555.30078125 cpu‚25.19553962096668 disk.sdb1.in‚0 disk.rbd2.in‚0 disk.dm-0.out‚58268.61328125 disk.sda.in‚ 4.30078125# proc.memory.rssMB‚ 2990.49609375 disk.sda1.in‚0 disk.md0p1.in‚ 5.12890625( proc.memory.availableMB‚ 652.71484375# disk.nvme1n1.out‚31407.85546875 disk.rbd2.out‚ 1.341796875) disk./.usagePercent‚75.86635998492758 disk.dm-0.in‚ 1102.05859375 disk.sdb.in‚0.828125 disk.sdb.out‚576.38818359375 network.sent‚ 1.0290322e+07 disk.sda2.in‚ 4.30078125 disk.rbd0.in‚0 disk.nvme1n1.in‚ 498.61328125 disk.sda1.out‚0 proc.cpu.threads‚17 disk.md0.out‚ 555.300781259iè8’³‚(:‡ ¸¨êÔî·íÿ gpu.0.gpu‚41.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚166.548' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚41.637 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0aæ”´‚(:‰ ǨêÔŸ‰œÿ gpu.0.gpu‚55.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚37.0 gpu.0.powerWatts‚161.091' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚40.27275 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0A-Âfnµ‚(:ã ǨêÔÌÛðÿ% disk./.usageGB‚2713.3485107421875 disk.sda.in‚ 4.30078125 disk.sdb2.in‚0.828125 disk.sda.out‚577.93798828125 disk.md0.out‚ 556.7890625 disk.sdb1.in‚0 disk.md0p1.out‚ 556.7890625 network.sent‚ 1.0400077e+07) disk./.usagePercent‚75.86635998492758! disk.sdb2.out‚577.93798828125 disk.md0p1.in‚ 5.12890625 cpu‚25.195001686319337 proc.cpu.threads‚17 network.recv‚ 1.650257e+06! disk.sda2.out‚577.93798828125 disk.rbd0.out‚ 1.404296875' proc.memory.availableMB‚ 652.8359375# proc.memory.rssMB‚ 2990.49609375 disk.sda1.in‚0 disk.md0.in‚ 5.12890625! disk.nvme0n1.out‚ 26860.765625 disk.sdb.out‚577.93798828125 disk.rbd2.out‚ 1.341796875$ memory_percent‚96.01540565490723 disk.rbd0.in‚0 disk.sda1.out‚0 disk.dm-0.in‚ 1102.05859375 disk.sdb.in‚0.828125 disk.rbd2.in‚0 disk.dm-0.out‚ 58268.78125 disk.nvme0n1.in‚ 603.4453125 disk.sda2.in‚ 4.30078125 disk.nvme1n1.in‚ 498.61328125! disk.nvme1n1.out‚ 31408.015625 disk.sdb1.out‚0) proc.memory.percent‚18.252539634704593Âe“¶‚(:ˆ Ö¨êÔÅøªÿ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚36.0 gpu.0.powerWatts‚114.131' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.53275 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‘i>ðr·‚(:ç Ö¨êÔ‡ÜÅÿ disk.rbd0.in‚0 disk.sda1.out‚0% proc.memory.availableMB‚ 652.84375) proc.memory.percent‚18.25253963470459 network.recv‚ 1.655034e+06 disk.sda.out‚579.85888671875 disk.dm-0.out‚58849.97265625) disk./.usagePercent‚75.85161213981165 disk.md0p1.in‚ 5.12890625# memory_percent‚96.0153579711914 cpu‚25.194526492991493$ disk./.usageGB‚2712.821056365967 proc.cpu.threads‚17 disk.sda.in‚ 4.30078125 disk.sda2.in‚ 4.30078125! disk.sda2.out‚579.85888671875 disk.nvme1n1.in‚ 498.61328125 disk.md0.in‚ 5.12890625 disk.md0.out‚ 558.65234375 disk.rbd2.in‚0 disk.rbd2.out‚ 1.341796875 disk.sdb1.out‚0! disk.sdb2.out‚579.85888671875# proc.memory.rssMB‚ 2990.49609375" disk.nvme0n1.out‚ 27091.7890625 disk.sdb.out‚579.85888671875# disk.nvme1n1.out‚31758.18359375 disk.sdb2.in‚0.828125 network.sent‚ 1.0404253e+07 disk.md0p1.out‚ 558.65234375 disk.sdb.in‚0.828125 disk.rbd0.out‚ 1.404296875 disk.sdb1.in‚0 disk.sda1.in‚0 disk.dm-0.in‚ 1102.05859375 disk.nvme0n1.in‚ 603.4453125Ó;…ᔸ‚(:‰ å¨êÔÆ·òý gpu.0.gpu‚56.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚39.0 gpu.0.powerWatts‚171.501' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.87525 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0$âtk¹‚(:à å¨êÔÕñêþ disk.sdb1.out‚0 disk.rbd2.in‚0 disk.rbd2.out‚ 1.341796875 disk.sdb2.in‚0.828125 disk.md0p1.in‚ 5.12890625" disk.nvme0n1.out‚ 27091.8046875 disk.sda.out‚581.35009765625' proc.memory.availableMB‚ 652.6953125 proc.cpu.threads‚17 network.recv‚ 1.658525e+06% disk./.usageGB‚2712.2656593322754 disk.rbd0.out‚ 1.404296875 disk.nvme1n1.in‚ 498.61328125 disk.sdb1.in‚0 disk.sda1.in‚0$ memory_percent‚96.01626396179199# proc.memory.rssMB‚ 2990.49609375 disk.dm-0.out‚ 58926.1875" disk.nvme1n1.out‚ 31834.3828125 disk.md0.in‚ 5.12890625) disk./.usagePercent‚75.83608300629791 disk.md0.out‚ 560.078125! disk.sdb2.out‚581.35009765625 disk.sda1.out‚0 disk.sdb.in‚0.828125 disk.sda2.in‚ 4.30078125! disk.sda2.out‚581.35009765625 disk.rbd0.in‚0 disk.dm-0.in‚ 1102.05859375 disk.md0p1.out‚ 560.078125 disk.nvme0n1.in‚ 603.4453125 disk.sdb.out‚581.35009765625 network.sent‚ 1.0409553e+07 disk.sda.in‚ 4.30078125) proc.memory.percent‚18.25253963470459 cpu‚25.1940579046988øŸÚŽ2º- _wandbruntime‚5055  _runtime‚5055E¬“öÅ»¿ % lr-AdamW/pg1‚0.000693208521452603 % lr-AdamW/pg2‚0.000693208521452603 trainer/global_step‚4349 % _timestamp‚1.7885154297359147e+09 _runtime‚5055.206226538 _step‚353xBDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚361477057646080 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚361476140235264 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚356542178616832 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚619 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5151640078848 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚81642893 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 trainer/global_step‚4349 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26090 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007241 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 108916284 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚55625813 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚617 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚89454118 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4934879029248 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚404190499754496 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚353557097733752 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚361477057646080 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407872 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 108942372 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177047040 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚399038859675648 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5151864744448 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411420 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214665 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955591168 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 224665600 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4933979621376 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚348641516519056 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1187 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 _runtime‚5055.207355833 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917410816 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26088 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚55625815 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚399038883615744 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 198370402 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407872 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚361476140235264 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 108942372 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 108916284 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚356542160613888 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4934879029248 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4933979621376 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚26017080 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚356542160613888 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚89454118 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4915581214696 _step‚354 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚89454120Ö*}û'¿õO `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚356542178616832 % _timestamp‚1.7885154297369056e+09 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚404190748360192 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚81642893 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 trainer/global_step‚4349 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26090 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007241 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 108916284 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚55625813 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚617 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚89454118 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4934879029248 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚404190499754496 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚353557097733752 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚361477057646080 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407872 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 108942372 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177047040 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚399038859675648 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5151864744448 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411420 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214665 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955591168 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 224665600 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4933979621376 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚348641516519056 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1187 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 _runtime‚5055.207355833 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917410816 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26088 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚55625815 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚399038883615744 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 198370402 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407872 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚361476140235264 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 108942372 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 108916284 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚356542160613888 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4934879029248 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4933979621376 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚26017080 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚356542160613888 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚89454118 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4915581214696 _step‚354 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚89454120 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1212 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4916478626116 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚361477057646080 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚361476140235264 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚356542178616832 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚619 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5151640078848 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 198441007 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚399128178896384 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚353637100940186 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899421184 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚616 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 108966989 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4916689876014 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 108940875 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31686 F@DeviceStatsMonitor.on_train_bs­¸×atch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚81660967 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1187 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4935092424704 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5153030446080 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚89474016 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 224652288 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚618 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 108940875 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚89474018 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 _runtime‚ 5056.17569839 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚26023394 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 248592384 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚404280984690176 trainer/global_step‚4349 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚55638191 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚348719513652544 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚26022778 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4917587287642 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26114 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚356621790909440 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4935092424704 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955591168 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 198414891 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 $ _timestamp‚1.788515430704584e+09 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚404281233282560 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607˜ÿ¦H'ÂýM E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4916689876014 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 108940875 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31686 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚81660967 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1187 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4935092424704 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5153030446080 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚89474016 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 224652288 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚618 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 108940875 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚89474018 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 _runtime‚ 5056.17569839 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚26023394 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 248592384 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚404280984690176 trainer/global_step‚4349 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚55638191 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚348719513652544 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚26022778 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4917587287642 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26114 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚356621790909440 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4935092424704 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955591168 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 198414891 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 $ _timestamp‚1.788515430704584e+09 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚404281233282560 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚356621808912384 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4935991845888 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 198441007 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚399128178896384 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚353637100940186 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899421184 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚616 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 108966989 FDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚360523675391488 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5209771604480 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4990533609472 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 110149186 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1212 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚357511549097260 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 $ _timestamp‚1.788515482313687e+09 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917410816 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955591168 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4990533609472 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214665 _runtime‚5107.784302115 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410816 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚365513309593088 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 110175274 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 398533632 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚357512463548040 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚56244615 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 200624294 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚56244613 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚352540521225456 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚365513309593088 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1187 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 240217088 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 trainer/global_step‚4399 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚82548873 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 110175274 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚360523675391488 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚82549475 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚26304860k>J×ò A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955591168 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚360523693394432 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚365514227003904 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407872 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚602 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450780 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚352540538264816 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4971925283224 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26090 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4989634201600 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564xõÉú'ÑôO R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚365513309593088 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1187 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 240217088 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 trainer/global_step‚4399 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚82548873 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 110175274 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚360523675391488 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚82549475 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚26304860 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955591168 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚360523693394432 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚365514227003904 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407872 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚602 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450780 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚352540538264816 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4971925283224 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26090 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4989634201600 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 _step‚358 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007241 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚90449020 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚408714592316928 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4971027871804 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 200598204 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚360523675391488 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5209771604480 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4990533609472 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 110149186 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1212 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚357511549097260 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 $ _timestamp‚1.788515482313687e+09 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917410816 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955591168 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4990533609472 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214665 _runtime‚5107.784302115 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410816 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚365513309593088 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 110175274 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 398533632 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚357512463548040 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚56244615 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 200624294 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚56244613 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚352540521225456 JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚90468918 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 110199891 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚360603323689984 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚360603323689984 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 110173777 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177047040 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚56256991 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450988 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4990747004928 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 200668809 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 200642693 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26114 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚598 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚82566803 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚596 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007241 _runtime‚5108.777705571 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4973033944750 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚365594970116096 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31686 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚26309814 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚90468918 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚365594052691968 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚403594139933184 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚352618518358944 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚360603305687040 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4991646426112 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 233040896 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411628 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚82567401 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚26310410 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4990747004928 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚90468916 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚360603305687040 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚357590637852706 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚365594970116096 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 110199891 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26114 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5211152542208 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 398533632 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4991646426112 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚90468916 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0QÕ€Ñ'ÔÿM FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31686 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚26309814 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚90468918 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚365594052691968 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚403594139933184 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚352618518358944 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚360603305687040 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4991646426112 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 233040896 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411628 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚82567401 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚26310410 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4990747004928 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚90468916 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚360603305687040 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚357590637852706 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚365594970116096 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 110199891 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26114 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5211152542208 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 398533632 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4991646426112 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚90468916 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177047040 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚56256989 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 trainer/global_step‚4399 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 200642693 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955591168 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 200668809 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚90468918 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 110199891 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚360603323689984 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚360603323689984 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 110173777 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177047040 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚56256991 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450988 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4990747004928 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 200668809 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 200642693 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26114 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚598 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚82566803 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚596 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007241 _runtime‚5108.777705571 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4973033944750 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚365594970116096çñkþ2Õ- _wandbruntime‚5108  _runtime‚5108èWn=zÖÕ 1loss/SRqqt1/detection_loss‚1.5282349659729813 .loss/FBt4/detection_loss‚2.763924648364385 trainer/global_step‚4399 _runtime‚5108.778754906 2loss/SRqqt1/assignment_loss‚1.2408749810025348 1loss/FBt4/assignment_loss‚0.19484783140288187 /loss/FBt2/detection_loss‚0.5592036936432123 %loss/total_loss‚41.94799972712196 0loss/FRt1/assignment_loss‚2.8281357796753155 0loss/FBt1/assignment_loss‚0.5178566642217837 /loss/FRt2/detection_loss‚1.5930408260651998 1loss/SRqqt2/assignment_loss‚1.004106963113431 0loss/FBt3/assignment_loss‚0.2971482554373021 1loss/SRqqt3/detection_loss‚1.7333762089411417 1loss/SRqqt4/detection_loss‚2.4212775374280997 epoch‚3 _step‚360 0loss/FRt2/assignment_loss‚0.8335490918585232 /loss/FRt4/assignment_loss‚3.928483247756958 /loss/FRt1/detection_loss‚1.9099616612463868 2losÐÂO ás/SRqqt3/assignment_loss‚1.4476966035034922 2loss/SRqqt4/assignment_loss‚2.0020539075906933 1loss/FBt2/assignment_loss‚0.07494032500038846 1loss/SRqqt2/detection_loss‚1.2001017351905185 -loss/FRt3/detection_loss‚2.45424644303758 /loss/FBt1/detection_loss‚3.2237353367464885 % _timestamp‚1.7885154833086333e+09 0loss/FRt3/assignment_loss‚2.7911763016770528 .loss/FRt4/detection_loss‚3.077215247154236 /loss/FBt3/detection_loss‚2.3228114710913763òÈ`×Ú 1loss/SRqqt4/detection_loss‚2.4212775374280997 epoch‚3 _step‚360 0loss/FRt2/assignment_loss‚0.8335490918585232 /loss/FRt4/assignment_loss‚3.928483247756958 /loss/FRt1/detection_loss‚1.9099616612463868 2loss/SRqqt3/assignment_loss‚1.4476966035034922 2loss/SRqqt4/assignment_loss‚2.0020539075906933 1loss/FBt2/assignment_loss‚0.07494032500038846 1loss/SRqqt2/detection_loss‚1.2001017351905185 -loss/FRt3/detection_loss‚2.45424644303758 /loss/FBt1/detection_loss‚3.2237353367464885 % _timestamp‚1.7885154833086333e+09 0loss/FRt3/assignment_loss‚2.7911763016770528 .loss/FRt4/detection_loss‚3.077215247154236 /loss/FBt3/detection_loss‚2.3228114710913763 1loss/SRqqt1/detection_loss‚1.5282349659729813 .loss/FBt4/detection_loss‚2.763924648364385 trainer/global_step‚4399 _runtime‚5108.778754906 2loss/SRqqt1/assignment_loss‚1.2408749810025348 1loss/FBt4/assignment_loss‚0.19484783140288187 /loss/FBt2/detection_loss‚0.5592036936432123 %loss/total_loss‚41.94799972712196 0loss/FRt1/assignment_loss‚2.8281357796753155 0loss/FBt1/assignment_loss‚0.5178566642217837 /loss/FRt2/detection_loss‚1.5930408260651998 1loss/SRqqt2/assignment_loss‚1.004106963113431 0loss/FBt3/assignment_loss‚0.2971482554373021 1loss/SRqqt3/detection_loss‚1.7333762089411417èµl)èžØ‚(:“ ¡©êÔ»Ñûý gpu.0.gpu‚56.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚37.0 gpu.0.powerWatts‚127.106' gpu.0.enforcedPowerLimitWatts‚400.0) gpu.0.powerPercent‚31.776499999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0J­2pÙ‚(:å ¡©êÔ‘Öáÿ disk.dm-0.in‚ 1102.05859375 disk.sdb.out‚588.2373046875 disk.rbd0.out‚ 1.404296875 proc.cpu.threads‚17) disk./.usagePercent‚75.84539233754151 disk.rbd2.in‚0 disk.sda1.in‚0 disk.sdb.in‚0.828125 disk.sda2.in‚ 4.30078125( proc.memory.availableMB‚ 652.59765625 network.sent‚ 1.0534787e+07 disk.dm-0.out‚59436.59765625$ memory_percent‚96.01686000823975 network.recv‚ 1.676617e+06 disk.md0.in‚ 5.12890625 disk.sdb2.in‚0.828125 disk.rbd0.in‚0 disk.nvme1n1.in‚ 498.61328125 disk.sdb1.in‚0 disk.rbd2.out‚ 1.341796875 disk.nvme0n1.in‚ 603.4453125# proc.memory.rssMB‚ 2990.49609375$ disk./.usageGB‚2712.598606109619 disk.md0p1.in‚ 5.12890625) proc.memory.percent‚18.25253963470459 cpu‚25.19207516460092 disk.sdb2.out‚588.2373046875 disk.sda1.out‚0# disk.nvme0n1.out‚27092.98828125 disk.sda.in‚ 4.30078125 disk.sda.out‚588.2373046875 disk.sda2.out‚588.2373046875 disk.sdb1.out‚0 disk.md0.out‚ 566.72265625 disk.md0p1.out‚ 566.72265625! disk.nvme1n1.out‚ 32343.609375 <ÀÓ”Ú‚(:‰ °©êÔœüçý gpu.0.gpu‚48.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚174.903' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.72575 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0çD„ÈcÛ‚(:Ø °©êÔÍ¥ãþ" disk.nvme0n1.out‚ 27093.1796875% disk./.usageGB‚2712.9123001098633 disk.sdb2.out‚ 589.69140625 disk.md0p1.out‚ 568.109375$ memory_percent‚96.01688385009766# proc.memory.rssMB‚ 2990.49609375 disk.rbd0.out‚ 1.404296875% proc.memory.availableMB‚ 652.59375 cpu‚25.19162331283119 disk.sdb1.out‚0 disk.sda1.in‚0 disk.sda2.in‚ 4.30078125) disk./.usagePercent‚75.85416335307947 disk.sda.in‚ 4.30078125 disk.sdb1.in‚0 disk.md0.out‚ 568.109375 disk.rbd2.in‚0 network.recv‚ 1.682638e+06# disk.nvme1n1.out‚32348.26953125 disk.sdb2.in‚0.828125 disk.dm-0.in‚ 1102.05859375 disk.dm-0.out‚59441.44921875 disk.rbd0.in‚0 disk.rbd2.out‚ 1.341796875 disk.sda2.out‚ 589.69140625 disk.nvme1n1.in‚ 498.61328125 network.sent‚ 1.0642252e+07 disk.sda1.out‚0 disk.md0p1.in‚ 5.12890625 disk.sdb.in‚0.828125 disk.sdb.out‚ 589.69140625 disk.sda.out‚ 589.69140625 disk.nvme0n1.in‚ 603.4453125 disk.md0.in‚ 5.12890625) proc.memory.percent‚18.25253963470459 proc.cpu.threads‚17è¢;2’Ü‚(:‡ ¿©êÔÌ˦ÿ gpu.0.gpu‚40.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚167.604' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚41.901 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0îÝ0vÝ‚(:ë ¿©êÔ†º¶€ disk.dm-0.in‚ 1102.05859375% disk./.usageGB‚2713.0986862182617# disk.nvme0n1.out‚27093.18359375 disk.sdb.out‚591.77978515625 disk.sda.out‚591.77978515625 disk.rbd0.out‚ 1.404296875 disk.nvme1n1.in‚ 498.61328125! disk.sdb2.out‚591.77978515625 disk.sda1.out‚0) proc.memory.percent‚18.25253963470459' proc.memory.availableMB‚ 652.3359375# proc.memory.rssMB‚ 2990.49609375 disk.rbd0.in‚0# disk.nvme1n1.out‚32563.83984375 disk.dm-0.out‚ 59657.0234375 cpu‚25.191195722140076 network.recv‚ 1.686683e+06) disk./.usagePercent‚75.85937478667894 disk.nvme0n1.in‚ 603.4453125 disk.sda.in‚ 4.30078125! disk.sda2.out‚591.77978515625 disk.md0.in‚ 5.12890625 disk.rbd2.in‚0 disk.rbd2.out‚ 1.341796875 disk.sdb2.in‚0.828125 disk.sda1.in‚0 disk.sdb.in‚0.828125 disk.md0.out‚ 570.14453125 disk.md0p1.out‚ 570.14453125$ memory_percent‚96.01845741271973 proc.cpu.threads‚17 disk.sda2.in‚ 4.30078125 disk.md0p1.in‚ 5.12890625 network.sent‚ 1.0648074e+07 disk.sdb1.in‚0 disk.sdb1.out‚0…(Û+”Þ‚(:‰ ΩêÔLj±ÿ gpu.0.gpu‚53.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚161.423' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚40.35575 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚08wÓÄjß‚(:ß ΩêÔ‹±Ü€ network.recv‚ 1.691324e+06 disk.sda2.in‚ 4.30078125 disk.sdb1.out‚0 disk.dm-0.out‚59959.37109375$ disk./.usageGB‚2713.349132537842 disk.rbd0.in‚0" disk.nvme1n1.out‚ 32866.1171875 disk.md0.out‚ 571.6171875 disk.sda1.in‚0) proc.memory.percent‚18.25253963470459 disk.nvme0n1.in‚ 603.4453125 disk.sda.out‚ 593.322265625 disk.rbd2.in‚0# disk.nvme0n1.out‚27093.25390625 disk.rbd0.out‚ 1.404296875# proc.memory.rssMB‚ 2990.49609375 network.sent‚ 1.0654387e+07 disk.nvme1n1.in‚ 498.61328125 disk.sdb2.out‚ 593.322265625 disk.sda1.out‚0( proc.memory.availableMB‚ 652.34765625 proc.cpu.threads‚17) disk./.usagePercent‚75.86637737059432 disk.sdb.in‚0.828125 disk.md0.in‚ 5.12890625 disk.rbd2.out‚ 1.341796875 disk.sdb2.in‚0.828125" memory_percent‚96.018385887146 disk.sdb.out‚ 593.322265625 disk.sda.in‚ 4.30078125 disk.sda2.out‚ 593.322265625 cpu‚25.190736073351307 disk.sdb1.in‚0 disk.dm-0.in‚ 1102.05859375 disk.md0p1.in‚ 5.12890625 disk.md0p1.out‚ 571.6171875,áÖD2à- _wandbruntime‚5162  _runtime‚5162òl øÅá¿ % lr-AdamW/pg1‚0.000683350127039282 % lr-AdamW/pg2‚0.000683350127039282 trainer/global_step‚4449 % _timestamp‚1.7885155370462239e+09 _runtime‚5162.516573537 _step‚361W$JÊâÄ trainer/global_step‚4449 % _timestamp‚1.7885155370462239e+09 _runtime‚5162.516573537 _step‚361 % lr-AdamW/pg1‚0.000683350127039282 % lr-AdamW/pg2‚0.000683350127039282éX–O2ã- _wandbruntime‚5162  _runtime‚5162†Sÿõ'äïO ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚364505208172032 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5046188189696 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚624 _runtime‚ 5162.51742023 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚369550478950912 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5268474737152 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚364505208172032 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 220471296 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚361466914911548 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1187 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 111408176 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5045288781824 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚56863413 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214665 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955591168 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚356439525931856 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917410816 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚364505190169088 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚369550478950912 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚369551396361728 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚91443920 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1212 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚369551396361728 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407872 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5268254265856 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚361466000460768 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚622 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚26591865 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5027371940332 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955591168 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 202852096 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5026474528912 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚56863415 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚83454656 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 202826006 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5045288781824 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26088 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚91443918 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177047040 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410816 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 244411392 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 111408176 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚83455280 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚413239280426496 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407872 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚83454656 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 202826006 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5045288781824 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26088 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚91443918 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177047040 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410816 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 244411392 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 111408176 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚83455280 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚413239280426496 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407872 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚369550478950912 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5268474737152 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚364505208172032 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 220471296 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚361466914911548 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1187 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 111408176 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5045288781824 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚56863413 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214665 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955591168 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚356439525931856 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917410816 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚364505190169088 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚369550478950912 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚369551396361728 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚91443920 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1212 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚369551396361728 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407872 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5268254265856 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚361466000460768 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚622 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚26591865 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5027371940332 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955591168 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 202852096 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5026474528912 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚56863415 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 JDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 202896611 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955591168 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚413329767656448 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5046401585152 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31686 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚615 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 526964274636¹ÉWfÝ8 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5046401585152 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411628 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5047301006336 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚617 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚356517523065344 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5027583190230 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚369632139473920 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917424128 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 $ _timestamp‚1.788515538063009e+09 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚369632139473920 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177047040 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚83473310 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚26597519 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1187 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚91463816 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899421184 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 202870495 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚364584820464640 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5047301006336 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚83472693 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚413329525355520 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚361546003667202 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 111432793 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚56875789 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 218360832 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 _runtime‚5163.533903311 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 202896611 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚91463818 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26114 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚364584838467584 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177047040 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 _step‚363 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 111432793iðA—'èþM ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5046401585152 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31686 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚615 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5269642746368 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5046401585152 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411628 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5047301006336 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚617 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚356517523065344 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5027583190230 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚369632139473920 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917424128 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 $ _timestamp‚1.788515538063009e+09 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚369632139473920 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177047040 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚83473310 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚26597519 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1187 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚91463816 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899421184 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 202870495 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚364584820464640 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5047301006336 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚83472693 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚413329525355520 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚361546003667202 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 111432793 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚56875789 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 218360832 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 _runtime‚5163.533903311 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 202896611 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚91463818 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26114 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚364584838467584 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177047040 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 _step‚363 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 111432793 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚364584838467584 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 111406679 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚56875791 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007241 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚369631222049792 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚26596904 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚408060100969984 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚356517506025984 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚91463818 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 202896611 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955591168 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚413329767656448 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0ëñ¨ x2é- _wandbruntime‚5163  _runtime‚5163Ó&YêÓ /loss/FRt3/assignment_loss‚2.500454118847847 2loss/SRqqt1/assignment_loss‚1.4390905718984348 0loss/FBt3/assignment_loss‚0.1761211242992431 /loss/FBt2/detection_loss‚0.5101575807871661 /loss/FRt1/assignment_loss‚2.900222660257266 0loss/FRt4/assignment_loss‚3.8048009509625644 0loss/FBt4/assignment_loss‚0.3409650500398129 /loss/FRt2/detection_loss‚1.3046816380810924 /loss/FBt3/detection_loss‚2.5474672582414417 1loss/FBt2/assignment_loss‚0.14320081145991748 /loss/FRt4/detection_loss‚3.2156095245610112 0loss/SRqqt3/detection_loss‚1.683443534009634 1loss/SRqqt4/detection_loss‚2.4719439510788237 trainer/global_step‚4449 2loss/SRqqt3/assignment_loss‚1.4405484302960179 1loss/SRqqt4/assignment_loss‚2.003373829115714 .loss/FRt1/detection_loss‚1.859475556294543 .loss/FBt4/detection_loss‚2.925481597582499 2loss/SRqqt2/assignment_loss‚1.3190752303720836 0loss/FBt1/assignment_loss‚0.1778028216212988 .loss/FRt3/detection_loss‚2.352848271280527 epoch‚3 % _timestamp‚1.7885155380652907e+09 1loss/SRqqt1/detection_loss‚1.5030971361058099 /loss/FBt1/detection_loss‚2.7328506443235607 %loss/total_loss‚41.86205145128176 _runtime‚5163.535422838 _step‚364 0loss/FRt2/assignment_loss‚1.5689536975696683 1loss/SRqqt2/detection_loss‚0.9403854621957768kIÅ^ëØ 2loss/SRqqt1/assignment_loss‚1.4390905718984348 0loss/FBt3/assignment_loss‚0.1761211242992431 /loss/FBt2/detection_loss‚0.5101575807871661 /loss/FRt1/assignment_loss‚2.900222660257266 0loss/FRt4/assignment_loss‚3.8048009509625644 0loss/FBt4/assignment_loss‚0.3409650500398129 /loss/FRt2/detection_loss‚1.3046816380810924 /loss/FBt3/detection_loss‚2.5474672582414417 1loss/FBt2/assignment_loss‚0.14320081145991748 /loss/FRt4/detection_loss‚3.2156095245610112 0loss/SRqqt3/detection_loss‚1.683443534009634 1loss/SRqqt4/detection_loss‚2.4719439510788237 trainer/global_step‚4449 2loss/SRqqt3/assignment_loss‚1.4405484302960179 1loss/SRqqt4/assignment_loss‚2.003373829115714 .loss/FRt1/detection_loss‚1.859475556294543 .loss/FBt4/detection_loss‚2.925481597582499 2loss/SRqqt2/assignment_loss‚1.3190752303720836 0loss/FBt1/assignment_loss‚0.1778028216212988 .loss/FRt3/detection_loss‚2.352848271280527 epoch‚3 % _timestamp‚1.7885155380652907e+09 1loss/SRqqt1/detection_loss‚1.5030971361058099 /loss/FBt1/detection_loss‚2.7328506443235607 %loss/total_loss‚41.86205145128176 _runtime‚5163.535422838 _step‚364 0loss/FRt2/assignment_loss‚1.5689536975696683 1loss/SRqqt2/detection_loss‚0.9403854621957768 /loss/FRt3/assignment_loss‚2.500454118847847ìòs¾Ý“ì‚(:ˆ Ý©êÔÙ̵ÿ gpu.0.gpu‚2.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚37.0 gpu.0.powerWatts‚163.763' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚40.94075 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0囊Ôrí‚(:ç Ý©êÔÊ™†‚ cpu‚25.190268540752925# disk.nvme1n1.out‚33214.79296875 disk.sda1.in‚0) disk./.usagePercent‚75.85160158041897 disk.md0.in‚ 5.12890625' proc.memory.availableMB‚ 652.5546875) proc.memory.percent‚18.25253963470459% disk./.usageGB‚2712.8206787109375 disk.sda.out‚594.8251953125 disk.sda2.in‚ 4.30078125 disk.md0p1.out‚ 573.05078125 disk.rbd0.out‚ 1.404296875 disk.nvme1n1.in‚ 498.61328125 disk.sdb1.out‚0 disk.rbd2.in‚0 disk.dm-0.in‚ 1102.05859375 disk.dm-0.out‚60308.08203125" disk.nvme0n1.out‚ 27093.2890625 disk.sdb.out‚594.8251953125 disk.sda.in‚ 4.30078125 disk.sda1.out‚0 disk.md0p1.in‚ 5.12890625 disk.nvme0n1.in‚ 603.4453125 proc.cpu.threads‚17 network.sent‚ 1.0672498e+07 disk.rbd0.in‚0 disk.md0.out‚ 573.05078125 disk.rbd2.out‚ 1.341796875$ memory_percent‚96.01712226867676 disk.sdb1.in‚0 disk.sdb2.in‚0.828125 disk.sdb2.out‚594.8251953125 network.recv‚ 1.695026e+06 disk.sda2.out‚594.8251953125 disk.sdb.in‚0.828125# proc.memory.rssMB‚ 2990.49609375ÜoJ°“î‚(:ˆ ì©êÔ´²ƒþ gpu.0.gpu‚32.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚158.746' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚39.6865 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0VF-~tï‚(:é ì©êÔ¡îÌÿ proc.cpu.threads‚17 disk.sdb.out‚596.87158203125 disk.rbd0.in‚0 disk.nvme0n1.in‚ 603.4453125 disk.sdb.in‚0.828125 network.recv‚ 1.70034e+06) disk./.usagePercent‚75.85159966052939 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.rbd2.in‚0 disk.sdb2.in‚0.828125! disk.sdb2.out‚596.87158203125 network.sent‚ 1.0767065e+07 disk.sda1.in‚0 disk.sda1.out‚0 disk.md0p1.out‚ 575.03515625# disk.nvme0n1.out‚27093.46484375 disk.sda.in‚ 4.30078125 disk.rbd0.out‚ 1.404296875 disk.dm-0.in‚ 1102.05859375 disk.sda.out‚596.87158203125# proc.memory.rssMB‚ 2990.49609375) proc.memory.percent‚18.25253963470459 disk.md0.in‚ 5.12890625 disk.dm-0.out‚ 60458.65625$ memory_percent‚96.01867198944092 cpu‚25.189794099919666 disk.sda2.in‚ 4.30078125! disk.sda2.out‚596.87158203125 disk.rbd2.out‚ 1.341796875( proc.memory.availableMB‚ 652.30078125% disk./.usageGB‚2712.8206100463867 disk.nvme1n1.in‚ 498.61328125# disk.nvme1n1.out‚33365.19140625 disk.md0.out‚ 575.03515625 disk.md0p1.in‚ 5.1289062594$Óð‚(:ˆ û©êÔ©Ÿ±ÿ gpu.0.gpu‚55.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚37.0 gpu.0.powerWatts‚124.994' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚31.2485 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0HnÞ`ñ‚(:Õ û©êÔÏ’Ó€# disk.nvme0n1.out‚27093.50390625 disk.nvme1n1.in‚ 498.61328125 disk.rbd2.in‚0 cpu‚25.18933468749152 network.recv‚ 1.704531e+06) disk./.usagePercent‚75.85103435970915 disk.md0p1.in‚ 5.12890625 disk.sdb.out‚ 598.638671875 disk.sdb.in‚0.828125 disk.sda2.out‚ 598.638671875 disk.rbd0.in‚0' proc.memory.availableMB‚ 652.5546875 disk.md0p1.out‚ 576.734375 disk.sda.in‚ 4.30078125 disk.sdb1.out‚0 disk.sdb2.out‚ 598.638671875 disk.sda2.in‚ 4.30078125 disk.sdb1.in‚0$ memory_percent‚96.01712226867676 network.sent‚ 1.0775197e+07 disk.sda.out‚ 598.638671875$ disk./.usageGB‚2712.800392150879 disk.sdb2.in‚0.828125 disk.dm-0.out‚60459.50390625 disk.nvme0n1.in‚ 603.4453125 disk.rbd0.out‚ 1.404296875 disk.nvme1n1.out‚33366 disk.md0.out‚ 576.734375 disk.rbd2.out‚ 1.341796875 disk.sda1.in‚0 disk.md0.in‚ 5.12890625# proc.memory.rssMB‚ 2990.49609375) proc.memory.percent‚18.25253963470459 proc.cpu.threads‚17 disk.sda1.out‚0 disk.dm-0.in‚ 1102.05859375/ð³2ò- _wandbruntime‚5218  _runtime‚5218BTN&ÇóÁ _runtime‚5218.594826656 _step‚365 & lr-AdamW/pg1‚0.0006783408573232377 & lr-AdamW/pg2‚0.0006783408573232377 trainer/global_step‚4499 % _timestamp‚1.7885155931241622e+09²Pr\ÌôÆ _step‚365 & lr-AdamW/pg1‚0.0006783408573232377 & lr-AdamW/pg2‚0.0006783408573232377 trainer/global_step‚4499 % _timestamp‚1.7885155931241622e+09 _runtime‚5218.594826656íkîš©2õ- _wandbruntime‚5218  _runtime‚5218•y Ð öðO [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5101842769920 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚417763414867968 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 % _timestamp‚1.7885155931252468e+09 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 112641078 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚623 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5100943362048 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚412436742786048 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450780 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 205053808 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚373588565719552 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚368486722949632 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26088 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5326894650368 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 205079898 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955591168 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 222568448 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214665 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5081921186020 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26088 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5101842769920 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚412436766726144 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26090 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚373587648308736 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚373588565719552 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚368486704946688 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚92438818 _runtime‚5218.595806181 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5082818597440 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411420 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 112614990 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚360338530638256 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407872 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.p'ñ(&eak‚ 13037993984 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177047040 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 112614990 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 246508544 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚365421366275056 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚84360214 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚621 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007241 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚57482213 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1187 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚365420451824276 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚373587648308736 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚92438820 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚26877999 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955591168 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0ÅÝW3û'÷õO OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5326672081920 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 112641078 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚373587648308736 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚373588565719552 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚368486704946688 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚92438818 _runtime‚5218.595806181 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5082818597440 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411420 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 112614990 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚360338530638256 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407872 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177047040 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 112614990 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 246508544 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚365421366275056 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚84360214 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚621 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007241 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚57482213 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1187 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚365420451824276 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚373587648308736 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚92438820 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚26877999 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955591168 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5101842769920 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚417763414867968 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 % _timestamp‚1.7885155931252468e+09 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 112641078 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚623 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5100943362048 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚412436742786048 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450780 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 205053808 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚373588565719552 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚368486722949632 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26088 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5326894650368 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 205079898 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955591168 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 222568448 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214665 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5081921186020 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26088 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5101842769920 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚412436766726144 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26090 JDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚84377656 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚368566335242240 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 112639581 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚613 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5327838373888 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 205098297 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26114 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31686 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411628 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26114 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917424128 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚57494589 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚4499 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 398533632 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5102955586560 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚92458718 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 112665695 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5102056165376 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 205124413 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚412526062006784 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5328063026176 % _timestamp‚1.7885155942948987e+09 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 205124413 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚611 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚57494591 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚412526085946880 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚368566335242240 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5102955586560 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1187 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5083927258966 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007241 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚417853900380672 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 248592384 _runtime‚5219.767006573 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚92458716 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955591168 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5102056165376 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177047040 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚84378269 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚368566353245184 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚373669308831744 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚365500455030710 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚373668391407616 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2œ¾8G'úÿM B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 205098297 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26114 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31686 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411628 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26114 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917424128 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚57494589 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚4499 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 398533632 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5102955586560 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚92458718 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 112665695 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5102056165376 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 205124413 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚412526062006784 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5328063026176 % _timestamp‚1.7885155942948987e+09 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 205124413 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚611 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚57494591 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚412526085946880 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚368566335242240 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5102955586560 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1187 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5083927258966 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007241 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚417853900380672 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 248592384 _runtime‚5219.767006573 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚92458716 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955591168 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5102056165376 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177047040 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚84378269 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚368566353245184 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚373669308831744 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚365500455030710 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚373668391407616 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899421184 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚417854148973056 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚365499540579722 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 112639581 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚84377656 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚368566335242240 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 112639581 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚613 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5327838373888 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984ïMf/2û- _wandbruntime‚5219  _runtime‚5219ßu‘ÞEüÔ 0loss/FRt2/assignment_loss‚1.3992810365946398 2loss/SRqqt3/assignment_loss‚1.4035195732845904 0loss/FBt1/assignment_loss‚0.4958842952520562 1loss/FBt2/assignment_loss‚0.08509528366507824 1loss/FBt3/assignment_loss‚0.24008810327650282 /loss/FBt2/detection_loss‚0.6126934823950227 %loss/total_loss‚43.19429621810251 2loss/SRqqt1/assignment_loss‚1.4257898864225815 .loss/FRt1/detection_loss‚2.157814096964912 1loss/SRqqt1/detection_loss‚1.4983928360171237 % _timestamp‚1.7885155942980762e+09 _step‚368 1loss/SRqqt2/detection_loÝlÕ3ss‚0.9829588306170923 1loss/SRqqt4/detection_loss‚2.6276239943504334 _runtime‚5219.768388119 /loss/FRt1/assignment_loss‚2.940437544223874 0loss/FRt3/assignment_loss‚2.9343116098926183 .loss/FRt4/assignment_loss‚3.37973889708519 1loss/FBt4/assignment_loss‚0.42236612279278535 1loss/SRqqt3/detection_loss‚2.0990727280682706 -loss/FBt1/detection_loss‚2.68287250825337 /loss/FBt3/detection_loss‚3.1071716427803038 /loss/FBt4/detection_loss‚2.3532506277163825 trainer/global_step‚4499 2loss/SRqqt4/assignment_loss‚1.6676372912526132 /loss/FRt2/detection_loss‚1.8866903111338615 .loss/FRt3/detection_loss‚2.456499977658192 .loss/FRt4/detection_loss‚3.310898560285568 epoch‚3 2loss/SRqqt2/assignment_loss‚1.0242069781194498%?p¢_ýÙ /loss/FRt1/assignment_loss‚2.940437544223874 0loss/FRt3/assignment_loss‚2.9343116098926183 .loss/FRt4/assignment_loss‚3.37973889708519 1loss/FBt4/assignment_loss‚0.42236612279278535 1loss/SRqqt3/detection_loss‚2.0990727280682706 -loss/FBt1/detection_loss‚2.68287250825337 /loss/FBt3/detection_loss‚3.1071716427803038 /loss/FBt4/detection_loss‚2.3532506277163825 trainer/global_step‚4499 2loss/SRqqt4/assignment_loss‚1.6676372912526132 /loss/FRt2/detection_loss‚1.8866903111338615 .loss/FRt3/detection_loss‚2.456499977658192 .loss/FRt4/detection_loss‚3.310898560285568 epoch‚3 2loss/SRqqt2/assignment_loss‚1.0242069781194498 0loss/FRt2/assignment_loss‚1.3992810365946398 2loss/SRqqt3/assignment_loss‚1.4035195732845904 0loss/FBt1/assignment_loss‚0.4958842952520562 1loss/FBt2/assignment_loss‚0.08509528366507824 1loss/FBt3/assignment_loss‚0.24008810327650282 /loss/FBt2/detection_loss‚0.6126934823950227 %loss/total_loss‚43.19429621810251 2loss/SRqqt1/assignment_loss‚1.4257898864225815 .loss/FRt1/detection_loss‚2.157814096964912 1loss/SRqqt1/detection_loss‚1.4983928360171237 % _timestamp‚1.7885155942980762e+09 _step‚368 1loss/SRqqt2/detection_loss‚0.9829588306170923 1loss/SRqqt4/detection_loss‚2.6276239943504334 _runtime‚5219.768388119ðeB\’þ‚(:‡ ŠªêÔó¥÷ý gpu.0.gpu‚2.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚37.0 gpu.0.powerWatts‚160.922' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚40.2305 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ä%†árÿ‚(:ç ŠªêÔ²Ÿ‚ disk.sdb.in‚0.828125 disk.sda2.in‚ 4.30078125! disk.sda2.out‚600.20849609375 disk.rbd2.in‚0 network.recv‚ 1.709193e+06 disk.sdb1.out‚0 disk.md0.out‚ 578.2421875 disk.md0p1.in‚ 5.12890625$ disk./.usageGB‚2712.831672668457# disk.nvme1n1.out‚33666.14453125 disk.md0p1.out‚ 578.2421875$ memory_percent‚96.01867198944092( proc.memory.availableMB‚ 652.30078125 proc.cpu.threads‚17 disk.sdb1.in‚0 disk.md0.in‚ 5.12890625! disk.nvme0n1.out‚ 27155.953125 disk.sda.in‚ 4.30078125 disk.sda.out‚600.20849609375 disk.sda1.in‚0 disk.rbd2.out‚ 1.341796875 disk.dm-0.in‚ 1102.05859375 disk.nvme0n1.in‚ 603.4453125# proc.memory.rssMB‚ 2990.49609375 cpu‚25.188872656174045 disk.sda1.out‚0 disk.dm-0.out‚60822.09765625) proc.memory.percent‚18.25253963470459! disk.sdb2.out‚600.20849609375 network.sent‚ 1.079212e+07) disk./.usagePercent‚75.85190897607255 disk.nvme1n1.in‚ 498.61328125 disk.rbd0.out‚ 1.404296875 disk.sdb2.in‚0.828125 disk.sdb.out‚600.20849609375 disk.rbd0.in‚0Љ©e”€‚(:‰ ™ªêÔ¹ª¦ÿ gpu.0.gpu‚57.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚37.0 gpu.0.powerWatts‚115.357' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.83925 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Râ«s‚(:è ™ªêÔÉ„üÿ# disk.nvme0n1.out‚27222.56640625 disk.nvme1n1.in‚ 498.61328125 disk.sdb2.out‚601.8935546875$ memory_percent‚96.02057933807373 disk.md0.in‚ 5.12890625) disk./.usagePercent‚75.83987852133347 disk.sda1.out‚0 disk.dm-0.in‚ 1102.05859375 disk.sdb.in‚0.828125 disk.sdb.out‚601.8935546875 cpu‚25.188356741347796 proc.cpu.threads‚17 disk.rbd0.out‚ 1.404296875 disk.sdb2.in‚0.828125( proc.memory.availableMB‚ 651.98828125 network.sent‚ 1.0888316e+07 disk.rbd2.in‚0# disk.nvme1n1.out‚33666.26953125 disk.sdb1.in‚0 disk.md0p1.out‚ 579.87109375 disk.sda.in‚ 4.30078125 disk.sda.out‚601.8935546875 disk.sda2.in‚ 4.30078125 disk.rbd0.in‚0) proc.memory.percent‚18.25253963470459 disk.rbd2.out‚ 1.341796875 disk.dm-0.out‚ 60888.8359375 disk.nvme0n1.in‚ 603.4453125 disk.sda2.out‚601.8935546875% disk./.usageGB‚2712.4014053344727 disk.sda1.in‚0 disk.md0p1.in‚ 5.12890625 disk.sdb1.out‚0# proc.memory.rssMB‚ 2990.49609375 disk.md0.out‚ 579.87109375 network.recv‚ 1.713679e+06×·W(“‚‚(:ˆ ¨ªêÔÜýÕý gpu.0.gpu‚56.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚39.0 gpu.0.powerWatts‚172.118' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.0295 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ãªóÔkƒ‚(:à ¨ªêÔ麵 disk.md0.in‚ 5.12890625 disk.md0p1.in‚ 5.12890625 disk.sda.out‚ 603.841796875 disk.sda2.out‚ 603.841796875# disk.nvme1n1.out‚33751.18359375 proc.cpu.threads‚17) disk./.usagePercent‚75.84336130768884 disk.rbd0.in‚0$ disk./.usageGB‚2712.525966644287 disk.md0.out‚ 581.76171875 disk.sdb2.in‚0.828125 disk.md0p1.out‚ 581.76171875 disk.sdb1.in‚0$ memory_percent‚96.02055549621582' proc.memory.availableMB‚ 651.9921875 network.recv‚ 1.71744e+06 disk.rbd2.in‚0 disk.sda1.in‚0 disk.sdb.out‚ 603.841796875 disk.rbd0.out‚ 1.404296875 disk.sdb1.out‚0# proc.memory.rssMB‚ 2990.49609375) proc.memory.percent‚18.25253963470459 disk.sdb2.out‚ 603.841796875 disk.dm-0.in‚ 1102.05859375 disk.sda.in‚ 4.30078125 disk.rbd2.out‚ 1.341796875 disk.sda1.out‚0! disk.nvme0n1.out‚ 27319.171875 disk.sdb.in‚0.828125 cpu‚25.187854081459786 network.sent‚ 1.0894158e+07 disk.dm-0.out‚61070.35546875 disk.nvme0n1.in‚ 603.4453125 disk.sda2.in‚ 4.30078125 disk.nvme1n1.in‚ 498.61328125sê{ ”„‚(:‰ ·ªêÔâ×äÿ gpu.0.gpu‚48.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚37.0 gpu.0.powerWatts‚133.017' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚33.25425 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0“Êœî[…‚(:Ð ·ªêÔ’ôã€( proc.memory.availableMB‚ 652.37890625 disk.nvme0n1.in‚ 603.4453125# disk.nvme1n1.out‚33751.35546875 disk.sdb1.in‚0 disk.sda2.in‚ 4.30078125 disk.rbd0.in‚0 disk.md0.in‚ 5.12890625# proc.memory.rssMB‚ 2990.49609375 disk.md0.out‚ 583.31640625 disk.rbd2.out‚ 1.341796875 disk.md0p1.in‚ 5.12890625 proc.cpu.threads‚17) disk./.usagePercent‚75.84336152100991% disk./.usageGB‚2712.5259742736816 disk.sda.in‚ 4.30078125 disk.md0p1.out‚ 583.31640625 disk.sdb.out‚ 605.46875$ memory_percent‚96.01819515228271 disk.sda.out‚ 605.46875 disk.nvme1n1.in‚ 498.61328125 disk.sdb1.out‚0 disk.sda1.in‚0 cpu‚25.187280696704015 network.recv‚ 1.7212e+06 disk.sdb2.in‚0.828125 disk.sda1.out‚0 disk.dm-0.out‚ 61155.578125) proc.memory.percent‚18.25253963470459 disk.rbd0.out‚ 1.404296875 network.sent‚ 1.089936e+07 disk.sda2.out‚ 605.46875 disk.rbd2.in‚0# disk.nvme0n1.out‚27404.22265625 disk.sdb2.out‚ 605.46875 disk.dm-0.in‚ 1102.05859375 disk.sdb.in‚0.828125K‘292†- _wandbruntime‚5271  _runtime‚5271d@FLJÁ & lr-AdamW/pg1‚0.0006732795941490347 & lr-AdamW/pg2‚0.0006732795941490347 trainer/global_step‚4549 % _timestamp‚1.7885156461584783e+09 _runtime‚5271.628947343 _step‚369¿2¯ÈÌˆÆ _runtime‚5271.628947343 _step‚369 & lr-AdamW/pg1‚0.0006732795941490347 & lr-AdamW/pg2‚0.0006732795941490347 trainer/global_step‚4549 % _timestamp‚1.7885156461584783e+09ñ®=¿‚2‰- _wandbruntime‚5271  _runtime‚5271ô.erö'ŠðO _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5384863684608 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚422287567507456 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410816 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚27164176 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 214179840 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚377625735077376 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚579 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚416902703822848 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 207307700 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26088 trainer/global_step‚4549 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 113847892 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚372468219724288 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚372468237727232 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1212 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚58101015 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5138265254548 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚422287805627392 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚377624817666560 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚372468237727232 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚58101013 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1187 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917410816 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 207281610 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407872 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚416902727762944 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚93433718 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 238119936 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5157497350144 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5156597942272 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407872 ÷ožû'‹õO P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214665 _runtime‚5271.630211839 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚93433718 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407872 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5384863684608 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚422287567507456 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410816 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚27164176 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 214179840 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚377625735077376 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚579 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚416902703822848 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 207307700 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26088 trainer/global_step‚4549 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 113847892 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚372468219724288 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚372468237727232 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1212 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚58101015 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5138265254548 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚422287805627392 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚377624817666560 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚372468237727232 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚58101013 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1187 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917410816 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 207281610 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407872 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚416902727762944 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚93433718 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 238119936 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5157497350144 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5156597942272 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 113898597 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1187 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5138476504446 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚416992023043584 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚364315515438784 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5139373916074 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚377706478189568 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚372547868022784 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚372547850019840 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 % _timestamp‚1.7885156472532427e+09 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚588 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚85282664 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚369453991943230 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚27169863 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 REDeviceStatsMonitor.on_train_ba³îÚßtch_end/inactive_split.large_pool.freed‚58113389 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450988 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _step‚371 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 113898597 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚377705560765440 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899421184 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 398533632 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5386022052864 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411628 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5158610166784 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚372547850019840 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚422378295785984 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 207326099 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚93453618 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 250689536 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955591168 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚27169275 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚85283254 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5157710745600 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 207352215 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899421184 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177047040 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5158610166784 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚93453616 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917424128 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 207352215 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚377706478189568 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚93453618 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚369454906394218 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚590 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚93453616 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26114 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 113872483 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 207326099 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 113872483 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚416992046983680$o'ŽÿM H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚590 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚93453616 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26114 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 113872483 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 207326099 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 113872483 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚416992046983680 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚364315532478144 _runtime‚5272.724287639 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5157710745600 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177047040 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚58113391 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 113898597 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1187 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5138476504446 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚416992023043584 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚364315515438784 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5139373916074 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚377706478189568 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚372547868022784 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚372547850019840 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 % _timestamp‚1.7885156472532427e+09 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚588 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚85282664 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚369453991943230 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚27169863 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚58113389 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450988 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _step‚371 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 113898597 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚377705560765440 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899421184 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 398533632 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5386022052864 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411628 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5158610166784 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚372547850019840 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚422378295785984 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 207326099 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚93453618 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 250689536 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955591168 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚27169275 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚85283254 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5157710745600 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 207352215 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899421184 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177047040 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5158610166784 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚93453616 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917424128 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 207352215 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚377706478189568 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚93453618 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚369454906394218 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171ó6Ï-!2- _wandbruntime‚5272  _runtime‚5272>ì€ZÔ 0loss/FRt3/assignment_loss‚2.5722026109695433 1loss/SRqqt1/assignment_loss‚1.270518599712151 1loss/FBt4/assignment_loss‚0.15340072107462926 0loss/FRt1/assignment_loss‚3.3589632883667946 1loss/FBt1/assignment_loss‚0.14164190785781253 1loss/SRqqt3/detection_loss‚1.3418972692452371 /loss/FBt2/detection_loss‚0.8964235534418433 2loss/SRqqt4/assignment_loss‚1.8692940574671542 1loss/SRqqt1/detection_loss‚1.6768490578582946 /loss/FBt1/detection_loss‚2.7126726458470025 /loss/FRt2/assignment_loss‚1.429271956605296 /loss/FRt4/assignment_loss‚4.119933681828635 0loss/FBt2/assignment_loss‚0.3296464235366687 .loss/FRt1/detection_loss‚2.477917062739531 %loss/total_loss‚43.29276266132108 epoch‚3 0loss/SRqqt4/detection_loss‚2.603843824139663 /loss/FBt4/detection_loss‚2.7367032965024314 trainer/global_step‚4549 _runtime‚5272.725491035 _step‚372 2loss/SRqqt2/assignment_loss‚0.9836684301597615 /loss/FRt2/detection_loss‚1.6758074708162778 -loss/FRt3/detection_loss‚2.20803707242012 1loss/SRqqt2/detection_loss‚1.0052760494490192 2loss/SRqqt3/assignment_loss‚1.5738669972633943 1loss/FBt3/assignment_loss‚0.30852522059447235 /loss/FRt4/detection_loss‚3.1338336041995456 .loss/FBt3/detection_loss‚2.712567859225803 % _timestamp‚1.7885156472553804e+09¬ý<'_‘Ù 1loss/SRqqt3/detection_loss‚1.3418972692452371 /loss/FBt2/detection_loss‚0.8964235534418433 2loss/SRqqt4/assignment_loss‚1.8692940574671542 1loss/SRqqt1/detection_loss‚1.6768490578582946 /loss/FBt1/detection_loss‚2.7126726458470025 /loss/FRt2/assignment_loss‚1.429271956605296 /loss/FRt4/assignment_loss‚4.119933681828635 0loss/FBt2/assignment_loss‚0.3296464235366687 .loss/FRt1/detection_loss‚2.477917062739531 %loss/total_loss‚43.29276266132108 epoch‚3 0loss/SRqqt4/detection_loss‚2.603843824139663 /loss/FBt4/detection_loss‚2.7367032965024314 trainer/global_step‚4549 _runtime‚5272.725491035 _step‚372 2loss/SRqqt2/assignment_loss‚0.9836684301597615 /loss/FRt2/detection_loss‚1.6758074708162778 -loss/FRt3/detection_loss‚2.20803707242012 1loss/SRqqt2/detection_loss‚1.0052760494490192 2loss/SRqqt3/assignment_loss‚1.5738669972633943 1loss/FBt3/assignment_loss‚0.30852522059447235 /loss/FRt4/detection_loss‚3.1338336041995456 .loss/FBt3/detection_loss‚2.712567859225803 % _timestamp‚1.7885156472553804e+09 0loss/FRt3/assignment_loss‚2.5722026109695433 1loss/SRqqt1/assignment_loss‚1.270518599712151 1loss/FBt4/assignment_loss‚0.15340072107462926 0loss/FRt1/assignment_loss‚3.3589632883667946 1loss/FBt1/assignment_loss‚0.14164190785781253ôEO&@k’‚(:à ƪêÔ³“ûþ disk.sda2.in‚ 4.30078125! disk.sda2.out‚607.01318359375 disk.rbd0.out‚ 1.404296875 disk.nvme1n1.in‚ 498.61328125 disk.dm-0.out‚61175.49609375 disk.md0p1.in‚ 5.12890625 cpu‚25.186816252373962( disk./.usagePercent‚75.8437964826599 disk.sdb1.out‚0 disk.sda1.in‚0 disk.nvme0n1.in‚ 603.4453125 disk.rbd2.in‚0 disk.sdb2.in‚0.828125 disk.md0.out‚ 584.796875 disk.sda1.out‚0 disk.sdb.in‚0.828125 disk.sda.out‚607.01318359375) proc.memory.percent‚18.25253963470459 network.recv‚ 1.725819e+06 disk.sdb.out‚607.01318359375& proc.memory.availableMB‚ 652.203125$ disk./.usageGB‚2712.541530609131 disk.rbd0.in‚0 disk.sdb1.in‚0 disk.md0p1.out‚ 584.796875# proc.memory.rssMB‚ 2990.49609375 disk.rbd2.out‚ 1.341796875! disk.sdb2.out‚607.01318359375 disk.dm-0.in‚ 1102.05859375# disk.nvme0n1.out‚27422.87109375$ memory_percent‚96.01926803588867 proc.cpu.threads‚17 network.sent‚ 1.0903435e+07 disk.nvme1n1.out‚ 33752.625 disk.md0.in‚ 5.12890625 disk.sda.in‚ 4.30078125î>Ë““‚(:ˆ ƪêÔþïõÿ gpu.0.gpu‚39.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚152.282' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚38.0705 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ŒFžþ’”‚(:‡ ÕªêÔ¯Ñ÷ý gpu.0.gpu‚51.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚161.864' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚40.466 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0\²Ep•‚(:å ÕªêÔô†‹ÿ disk.rbd2.out‚ 1.341796875 disk.sdb2.in‚0.828125 disk.sdb2.out‚609.0439453125 disk.sda.out‚609.0439453125 disk.rbd2.in‚0 disk.sda1.in‚0 disk.dm-0.out‚61401.74609375 disk.sdb.in‚0.828125# disk.nvme1n1.out‚33752.79296875 network.recv‚ 1.730144e+06 disk.md0p1.in‚ 5.12890625! disk.nvme0n1.out‚ 27648.953125 disk.nvme1n1.in‚ 498.61328125 disk.md0.in‚ 5.12890625 disk.md0.out‚ 586.75390625 disk.nvme0n1.in‚ 603.4453125) disk./.usagePercent‚75.84482170369465$ disk./.usageGB‚2712.578197479248 disk.sda2.in‚ 4.30078125$ memory_percent‚96.01864814758301 disk.sda1.out‚0 disk.dm-0.in‚ 1102.05859375 disk.md0p1.out‚ 586.75390625 disk.rbd0.in‚0 disk.sdb1.in‚0 disk.sdb1.out‚0 cpu‚25.186382381939357 proc.cpu.threads‚17 network.sent‚ 1.1010165e+07 disk.sda2.out‚609.0439453125# proc.memory.rssMB‚ 2990.49609375) proc.memory.percent‚18.25253963470459 disk.sdb.out‚609.0439453125 disk.sda.in‚ 4.30078125 disk.rbd0.out‚ 1.404296875' proc.memory.availableMB‚ 652.3046875 à( ’–‚(:‡ äªêÔí§êÿ gpu.0.gpu‚28.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚37.0 gpu.0.powerWatts‚157.852' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚39.463 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ø*ÿél—‚(:á äªêÔòÏÞ€) proc.memory.percent‚18.25253963470459% disk./.usageGB‚2712.3095092773438 disk.sdb.in‚0.828125 disk.rbd0.in‚0 disk.md0.out‚588.4375 disk.rbd2.out‚ 1.341796875 disk.md0p1.in‚ 5.12890625! disk.nvme0n1.out‚ 27757.421875 disk.sdb.out‚610.79736328125 disk.sda.out‚610.79736328125 disk.md0.in‚ 5.12890625 disk.sdb2.in‚0.828125 disk.sda1.in‚0 disk.sda1.out‚0 network.sent‚ 1.1016279e+07 disk.md0p1.out‚588.4375! disk.sda2.out‚610.79736328125 disk.rbd0.out‚ 1.404296875" disk.nvme1n1.out‚ 33753.4453125 disk.sdb1.in‚0 disk.sdb1.out‚0! disk.sdb2.out‚610.79736328125 proc.cpu.threads‚17 disk.dm-0.in‚ 1102.05859375 disk.sda.in‚ 4.30078125 disk.nvme1n1.in‚ 498.61328125 network.recv‚ 1.734668e+06) disk./.usagePercent‚75.83730906911465 disk.dm-0.out‚ 61510.8671875 disk.rbd2.in‚0$ memory_percent‚96.01924419403076 cpu‚25.185935787333214 disk.nvme0n1.in‚ 603.4453125 disk.sda2.in‚ 4.30078125( proc.memory.availableMB‚ 652.20703125# proc.memory.rssMB‚ 2990.49609375¯©ö“˜‚(:ˆ óªêÔ²æÉÿ gpu.0.gpu‚3.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚155.527' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚38.88175 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0žsQþv™‚(:ë óªêÔ†Æ) disk./.usagePercent‚75.84490948531258 disk.rbd0.out‚ 1.404296875 disk.md0.in‚ 5.12890625 disk.sdb2.in‚0.828125 disk.sda1.in‚0 disk.rbd0.in‚0 disk.sdb1.out‚0 disk.rbd2.out‚ 1.341796875! disk.sdb2.out‚612.38330078125 disk.nvme0n1.in‚ 603.4453125! disk.sda2.out‚612.38330078125 disk.sda1.out‚0 disk.dm-0.in‚ 1102.05859375 disk.dm-0.out‚61565.65234375# proc.memory.rssMB‚ 2990.49609375 proc.cpu.threads‚17 disk.sda.in‚ 4.30078125 disk.sdb1.in‚0 disk.sda2.in‚ 4.30078125 disk.nvme1n1.in‚ 498.61328125 disk.rbd2.in‚0 disk.sdb.in‚0.828125 disk.sdb.out‚612.38330078125% disk./.usageGB‚2712.5813369750977 disk.md0p1.out‚ 589.94921875$ memory_percent‚96.01776599884033) proc.memory.percent‚18.25253963470459 cpu‚25.185504417406346 network.sent‚ 1.102202e+07 network.recv‚ 1.738855e+06# disk.nvme1n1.out‚33753.61328125 disk.md0p1.in‚ 5.12890625 disk.sda.out‚612.38330078125 disk.md0.out‚ 589.94921875" disk.nvme0n1.out‚ 27812.0390625( proc.memory.availableMB‚ 652.44921875pħÿ2š- _wandbruntime‚5325  _runtime‚5325ÒÅÆ›À trainer/global_step‚4599 $ _timestamp‚1.788515700466557e+09 _runtime‚5325.936981585 _step‚373 & lr-AdamW/pg1‚0.0006681673922763327 & lr-AdamW/pg2‚0.0006681673922763327ïë~ݘŠ$ _timestamp‚1.788515700466557e+09 _runtime‚5325.936981585 _step‚373 & lr-AdamW/pg1‚0.0006681673922763327 & lr-AdamW/pg2‚0.0006681673922763327 trainer/global_step‚4599õ‹Íå2- _wandbruntime‚5325  _runtime‚5325rÆ ¥’žïO LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 115080794 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚381662904435200 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 trainer/global_step‚4599 _step‚374 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚376449752504832 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5443508000768 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450780 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚27452165 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5212252522496 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚373330269002072 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 % _timestamp‚1.7885157004673674e+09 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚426812196800512 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214665 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚368136540051056 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚86171980 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 246508544 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 209509412 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 115106882 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5213151930368 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚94428618 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 115080794 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411420 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚381661987024384 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚94428620 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚426811950291968 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚421368664859648 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5212252522496 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5193711911656 _runtime‚ 5325.93789853 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚381661987024384 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 209535502 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955591168 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚58719815 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚94428618 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚376449752504832 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407872 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1212 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚421368688799744 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5443285432320 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917410816 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚619 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚86171361 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚376449734501888 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177047040hê9ú'ŸôO JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚373329354551292 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407872 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚27451548 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚58719813 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚94428620 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 115106882 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007241 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411420 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚381661987024384 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚94428620 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚426811950291968 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚421368664859648 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5212252522496 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5193711911656 _runtime‚ 5325.93789853 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚381661987024384 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 209535502 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955591168 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚58719815 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚94428618 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚376449752504832 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407872 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1212 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚421368688799744 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5443285432320 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917410816 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚619 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚86171361 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚376449734501888 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177047040 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 115080794 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚381662904435200 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 trainer/global_step‚4599 _step‚374 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚376449752504832 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5443508000768 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450780 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚27452165 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5212252522496 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚373330269002072 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 % _timestamp‚1.7885157004673674e+09 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚426812196800512 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214665 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚368136540051056 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚86171980 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 246508544 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 209509412 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 115106882 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5213151930368 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚94428618 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 115080794 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 209580017 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 115131499 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450988 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚376529364797440 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚94448518 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5193923161554 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007241 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 209580017 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 209553901 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚426902423074304 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411628 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 115131499 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚376529382800384 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚27457215 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚421457984080384 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚86189404 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚368214537184544 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5214264747008 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 398533632 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564­îõO"¢þM MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26114 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5444669937664 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 209580017 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 115131499 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450988 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚376529364797440 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚94448518 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5193923161554 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007241 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 209580017 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 209553901 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚426902423074304 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411628 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 115131499 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚376529382800384 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚27457215 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚421457984080384 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚86189404 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚368214537184544 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5214264747008 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 398533632 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917424128 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚376529382800384 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚27457827 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚612 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚86190018 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚58732191 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31686 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5444438993920 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚94448516 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899421184 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5213365325824 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1187 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚94448518 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚381743647547392 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 254883840 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚421458008020480 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5194820573182 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899421184 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177047040 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 PDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407872 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411420 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007241 $ _timestamp‚1.788515757963484e+09 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 211737214 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 116313696 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚385699156382208 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177047040 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚380431249279488 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚95423520 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26088 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5267907102720 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚27739273 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚425834649836544 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚385699156382208 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214665 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917410816 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5268806510592 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 211763304 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410816 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚385700073793024 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5248261157344 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚380431267282432 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5268806510592 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 238119936 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚95423518 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚425834625896448 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚95423520 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚95423518 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚27738675 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 214179840 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚59338613 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407872 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 trainer/global_step‚4649 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 116339784pa¯ú'±ôO LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 116339784 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚380431267282432 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚87077288 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 116313696 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚87077888 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚377284720365580 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚372035561796816 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5267907102720 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚600 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 398533632 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955591168 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5501528781312 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 _step‚378 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚385700073793024 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 211737214 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 116313696 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚385699156382208 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177047040 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚380431249279488 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚95423520 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26088 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5267907102720 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚27739273 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚425834649836544 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚385699156382208 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214665 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917410816 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5268806510592 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 211763304 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410816 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚385700073793024 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5248261157344 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚380431267282432 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5268806510592 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 238119936 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚95423518 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚425834625896448 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚95423520 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚95423518 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚27738675 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 214179840 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚59338613 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407872 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 trainer/global_step‚4649 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 116339784 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚372035544757456 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5249158568764 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚380431249279488 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26090 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5501742961152 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407872 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411420 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007241 $ _timestamp‚1.788515757963484e+09 JDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚95443416 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917424128 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚59350991 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚95443418 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 116338287 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚59350989 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚372113541890944 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 trainer/global_step‚4649 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 398533632 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 211807819 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚380510879575040 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚95443416 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 116338287 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5249369818662 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚87095369 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899421184 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955591168 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚377362894670246 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _step‚379 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚372113524851584 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚431426886962176 $ _timestamp‚1.788515758981082e+09 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚380510897577984 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177047040 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5269019906048 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚380510897577984 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚27744380 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26114 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 116364401 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚95443418 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚593 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚425923969057280 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚377363809121234 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411628 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007241 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5269919327232 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899421184 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚425923945117184 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 211781703 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚431426642564096 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚385780816905216 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚385780816905216 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5502697446912 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5269019906048 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177047040 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 116364401 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚27744973 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 211807819 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1187 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450988 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚87095964 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944"ç/í'´þM OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917424128 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚59350991 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚95443418 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 116338287 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚59350989 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚372113541890944 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 trainer/global_step‚4649 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 398533632 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 211807819 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚380510879575040 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚95443416 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 116338287 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5249369818662 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚87095369 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899421184 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955591168 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚377362894670246 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _step‚379 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚372113524851584 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚431426886962176 $ _timestamp‚1.788515758981082e+09 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚380510897577984 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177047040 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5269019906048 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚380510897577984 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚27744380 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26114 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 116364401 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚95443418 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚593 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚425923969057280 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚377363809121234 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411628 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007241 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5269919327232 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899421184 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚425923945117184 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 211781703 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚431426642564096 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚385780816905216 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚385780816905216 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5502697446912 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5269019906048 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177047040 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 116364401 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚27744973 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 211807819 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1187 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450988 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚87095964 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 211781703 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚95443416û=‡2µ- _wandbruntime‚5384  _runtime‚5384Hz6[¶Õ 0loss/FBt3/assignment_loss‚0.3931784533762968 1loss/SRqqt3/assignment_loss‚1.198801228683442 2loss/SRqqt4/assignment_loss‚1.7863931348142417 .loss/FBt1/detection_loss‚2.006459057331085 %loss/total_loss‚40.80058213954859 /loss/FRt3/assignment_loss‚2.554650669195214 2loss/SRqqt2/assignment_loss‚1.1387781767943312 /loss/FRt3/detection_loss‚2.1750874596895002 1loss/SRqqt1/detection_loss‚1.8575380893082034 1loss/SRqqt4/detection_loss‚2.6600471657255422 /loss/FBt3/detection_loss‚2.3869789242744446 _runtime‚5384.453750985 _step‚380 1loss/SRqqt2/detection_loss‚1.0633755375350278 1loss/SRqqt3/detection_loss‚1.4769577557841937 /loss/FRt4/assignment_loss‚3.346889240401132 1loss/FBt2/assignment_loss‚0.15184405146135596 trainer/global_step‚4649 0loss/FBt1/assignment_loss‚0.7072954469070302 1loss/FBt4/assignment_loss‚0.11962610642258369 .loss/FRt2/detection_loss‚1.963087705274423 /loss/FBt2/detection_loss‚0.5462904748482964 /loss/FBt4/detection_loss‚2.3841503739356993 % _timestamp‚1.7885157589836173e+09 0loss/FRt1/assignment_loss‚2.7781069175555158 0loss/FRt2/assignment_loss‚1.8061721987194486 1loss/SRqqt1/assignment_loss‚1.452672969701947 /loss/FRt1/detection_loss‚1.9123572401271192 /loss/FRt4/detection_loss‚2.9338437616825104 epoch‚3Ô¥¹`·Ú /loss/FBt4/detection_loss‚2.3841503739356993 % _timestamp‚1.7885157589836173e+09 0loss/FRt1/assignment_loss‚2.7781069175555158 0loss/FRt2/assignment_loss‚1.8061721987194486 1loss/SRqqt1/assignment_loss‚1.452672969701947 /loss/FRt1/detection_loss‚1.9123572401271192 /loss/FRt4/detection_loss‚2.9338437616825104 epoch‚3 0loss/FBt3/assignment_loss‚0.3931784533762968 1loss/SRqqt3/assignment_loss‚1.198801228683442 2loss/SRqqt4/assignment_loss‚1.7863931348142417 .loss/FBt1/detection_loss‚2.006459057331085 %loss/total_loss‚40.80058213954859 /loss/FRt3/assignment_loss‚2.554650669195214 2loss/SRqqt2/assignment_loss‚1.1387781767943312 /loss/FRt3/detection_loss‚2.1750874596895002 1loss/SRqqt1/detection_loss‚1.8575380893082034 1loss/SRqqt4/detection_loss‚2.6600471657255422 /loss/FBt3/detection_loss‚2.3869789242744446 _runtime‚5384.453750985 _step‚380 1loss/SRqqt2/detection_loss‚1.0633755375350278 1loss/SRqqt3/detection_loss‚1.4769577557841937 /loss/FRt4/assignment_loss‚3.346889240401132 1loss/FBt2/assignment_loss‚0.15184405146135596 trainer/global_step‚4649 0loss/FBt1/assignment_loss‚0.7072954469070302 1loss/FBt4/assignment_loss‚0.11962610642258369 .loss/FRt2/detection_loss‚1.963087705274423 /loss/FBt2/detection_loss‚0.5462904748482964üR±l¸‚(:á ¯«êԺŲÿ network.sent‚ 1.1147018e+07 disk.nvme1n1.in‚ 498.61328125 disk.sdb1.out‚0 network.recv‚ 1.756229e+06$ disk./.usageGB‚2713.081985473633 disk.sdb1.in‚0 disk.md0.in‚ 5.12890625 disk.sdb.out‚619.1083984375( disk./.usagePercent‚75.8589078268693 disk.rbd0.out‚ 1.404296875 disk.sdb2.out‚619.1083984375# proc.memory.rssMB‚ 2990.49609375 proc.cpu.threads‚17 disk.md0.out‚ 596.4296875 disk.sdb.in‚0.828125 disk.sda2.out‚619.1083984375$ memory_percent‚96.01929187774658 disk.rbd0.in‚0 disk.sda.out‚619.1083984375 disk.sda1.in‚0" disk.nvme0n1.out‚ 28362.5546875 disk.sda2.in‚ 4.30078125 disk.md0p1.in‚ 5.12890625 disk.nvme0n1.in‚ 603.4453125 disk.sda.in‚ 4.30078125" disk.nvme1n1.out‚ 34016.3984375 disk.rbd2.in‚0 disk.sdb2.in‚0.828125 disk.sda1.out‚0 disk.dm-0.out‚ 62378.953125 disk.rbd2.out‚ 1.341796875 disk.dm-0.in‚ 1102.05859375 disk.md0p1.out‚ 596.4296875( proc.memory.availableMB‚ 652.19921875) proc.memory.percent‚18.25253963470459 cpu‚25.183425574539108³ˆs»”¹‚(:‰ ¯«êÔŽâ„€ gpu.0.gpu‚49.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚166.825' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚41.70625 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Añ;’º‚(:‡ ¾«êÔõˆºÿ gpu.0.gpu‚44.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚164.372' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚41.093 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0XV§®Z»‚(:Ï ¾«êÔËÕÿ disk.md0p1.out‚ 598.2734375 disk.rbd0.out‚ 1.404296875# proc.memory.rssMB‚ 2990.52734375 disk.sda1.out‚0 disk.md0p1.in‚ 5.12890625 disk.sda.in‚ 4.30078125 disk.sda2.in‚ 4.30078125 disk.sda2.out‚ 621.03125 disk.rbd0.in‚0( proc.memory.availableMB‚ 655.87890625 disk.sdb2.in‚0.828125 disk.dm-0.out‚ 62468.1484375 disk.nvme0n1.in‚ 603.4453125 disk.sdb1.in‚0 disk.md0.in‚ 5.12890625 disk.md0.out‚ 598.2734375$ memory_percent‚95.99683284759521 cpu‚25.1830000522655 network.recv‚ 1.76227e+06 disk.sdb.in‚0.828125 disk.sda.out‚ 621.03125) disk./.usagePercent‚75.85890526701654 disk.nvme1n1.in‚ 498.61328125 disk.rbd2.in‚0 disk.rbd2.out‚ 1.341796875 disk.sdb2.out‚ 621.03125# disk.nvme1n1.out‚34105.42578125 network.sent‚ 1.1255047e+07 disk.sda1.in‚0 disk.dm-0.in‚ 1102.05859375 disk.sdb.out‚ 621.03125) proc.memory.percent‚18.25273036956787% disk./.usageGB‚2713.0818939208984# disk.nvme0n1.out‚28362.72265625 disk.sdb1.out‚0 proc.cpu.threads‚17¶<¾ Y¼‚(:Î Í«êÔ­ÕÑÿ cpu‚25.182587620966267# disk.nvme1n1.out‚34238.60546875 disk.sda1.in‚0 disk.dm-0.in‚ 1102.05859375 disk.md0p1.in‚ 5.12890625 disk.sdb.in‚0.828125 disk.rbd2.in‚0 disk.md0p1.out‚ 599.87109375 disk.sdb.out‚622.6875 disk.sda.in‚ 4.30078125 disk.md0.in‚ 5.12890625# proc.memory.rssMB‚ 2990.52734375 disk.rbd0.in‚0 disk.sdb2.in‚0.828125 disk.dm-0.out‚ 62901.28125 disk.nvme0n1.in‚ 603.4453125$ memory_percent‚95.99683284759521 network.sent‚ 1.1260307e+07 disk.md0.out‚ 599.87109375# disk.nvme0n1.out‚28662.67578125 network.recv‚ 1.765545e+06 disk.sda2.out‚622.6875 disk.rbd0.out‚ 1.404296875 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.rbd2.out‚ 1.341796875 disk.sdb2.out‚622.6875% disk./.usageGB‚2713.0819511413574 disk.sda.out‚622.6875 disk.sda2.in‚ 4.30078125 disk.nvme1n1.in‚ 498.61328125 disk.sda1.out‚0( proc.memory.availableMB‚ 655.87890625) proc.memory.percent‚18.25273036956787 proc.cpu.threads‚17) disk./.usagePercent‚75.858906866924515²`”½‚(:‰ Í«êÔ®êÞÿ gpu.0.gpu‚46.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚164.989' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚41.24725 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0R }”¾‚(:‰ Ü«êÔÊïåÿ gpu.0.gpu‚56.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚173.953' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.48825 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0£=;ço¿‚(:ä Ü«êÔ¤âÿÿ disk.md0p1.in‚ 5.12890625 cpu‚25.182127897083717 network.recv‚ 1.769874e+06) disk./.usagePercent‚75.83577187752573 disk.sda.in‚ 4.30078125 disk.sda1.in‚0 disk.sdb.out‚624.1591796875% disk./.usageGB‚2712.2545318603516 disk.sdb2.out‚624.1591796875 disk.dm-0.in‚ 1102.05859375 disk.nvme0n1.in‚ 603.4453125$ memory_percent‚95.99745273590088 disk.nvme1n1.in‚ 498.61328125 disk.sda2.out‚624.1591796875 disk.rbd0.in‚0 disk.rbd2.in‚0 disk.rbd2.out‚ 1.341796875( proc.memory.availableMB‚ 655.77734375# proc.memory.rssMB‚ 2990.52734375 disk.dm-0.out‚62901.78515625# disk.nvme0n1.out‚28662.85546875 disk.rbd0.out‚ 1.404296875 network.sent‚ 1.1268527e+07 disk.sda2.in‚ 4.30078125 disk.md0p1.out‚ 601.296875 disk.sdb.in‚0.828125 proc.cpu.threads‚17 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.sda1.out‚0) proc.memory.percent‚18.25273036956787" disk.nvme1n1.out‚ 34238.9296875 disk.sda.out‚624.1591796875 disk.md0.in‚ 5.12890625 disk.md0.out‚ 601.296875 disk.sdb2.in‚0.828125I¶å"2À- _wandbruntime‚5438  _runtime‚5438¾Å(ÇÁÁ _runtime‚5438.400206416 _step‚381 & lr-AdamW/pg1‚0.0006577944443297888 & lr-AdamW/pg2‚0.0006577944443297888 trainer/global_step‚4699 % _timestamp‚1.7885158129298565e+09}·Ý;ÌÂÆ _runtime‚5438.400206416 _step‚381 & lr-AdamW/pg1‚0.0006577944443297888 & lr-AdamW/pg2‚0.0006577944443297888 trainer/global_step‚4699 % _timestamp‚1.7885158129298565e+09ý=Á|h2Ã- _wandbruntime‚5438  _runtime‚5438ÌZóÃ}ÄïO \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚389737243150848 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5323561682944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚389736325740032 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5559867843584 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407872 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚384412782060032 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007241 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚96418418 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410816 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26088 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214665 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚375934566503216 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 213965016 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 117572686 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚389736325740032 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5304605225872 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 213991106 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26088 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚96418418 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚87983078 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 _runtime‚5438.401222351 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚632 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚634 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1187 trainer/global_step‚4699 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚384412782060032 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955591168 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚384412764057088 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚59957415 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 117546598 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚87983712 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1212 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚430300586933248 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955591168 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5324461090816 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450780 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚381238257278308 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚96418420 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5324461090816 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917410816 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5323561682944 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚59957413 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚435860454776832 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚430300610873344 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 235151360 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚381239171729088 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177047040 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 _step‚382 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407872 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚96418420 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5324461090816 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917410816 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5323561682944 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚59957413 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚435860454776832 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚430300610873344 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 235151360 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚381239171729088 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177047040 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 _step‚382 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407872 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 213965016 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 117572686 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚389736325740032 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5304605225872 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 213991106 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26088 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚96418418 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚87983078 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 _runtime‚5438.401222351 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚632 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚634 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1187 trainer/global_step‚4699 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚384412782060032 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955591168 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚384412764057088 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚59957415 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 117546598 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚87983712 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1212 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚430300586933248 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955591168 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5324461090816 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450780 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚381238257278308 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚375934549463856 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411420 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚384412764057088 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712þqS·¥2Æ- _wandbruntime‚5439  _runtime‚5439ñ²'F'ÇúM JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177047040 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1187 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 117571189 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚96438318 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 214035621 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚88001220 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 230943744 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚384492412355584 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚384492412355584 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450988 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚376012546597344 _step‚383 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 214009505 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚28032075 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 254883840 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 398533632 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5325573907456 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚389817986263040 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚59969791 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 117571189 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚96438316 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 214035621 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚389817068838912 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 117597303 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5325573907456 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 117597303 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚88001866 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚389817986263040 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚384492394352640 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917424128 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚644 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 _runtime‚5439.351674566 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 214009505 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955591168 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚96438318 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚435950924610048 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0nç 9!ÈÿM ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5561018456064 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177047040 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5305713887398 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411628 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚389817068838912 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 117597303 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5325573907456 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 117597303 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚88001866 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚389817986263040 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚384492394352640 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917424128 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚644 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 _runtime‚5439.351674566 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 214009505 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955591168 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚96438318 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚435950924610048 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177047040 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1187 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 117571189 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚96438318 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 214035621 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚88001220 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 230943744 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚384492412355584 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚384492412355584 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450988 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚376012546597344 _step‚383 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 214009505 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚28032075 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 254883840 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 398533632 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5325573907456 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚389817986263040 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚59969791 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 117571189 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚96438316 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 214035621 FDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955591168 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 216218908 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚88891338 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1212 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411420 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚393773495097856 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚388394278834688 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚97413318 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚393773495097856 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410816 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5380115671040 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚611 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚434766571910144 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 118779500 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5380115671040 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚388394296837632 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26088 _step‚386 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚28315123 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5618177637888 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5379216263168 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚88890727 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5618406497792 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5359154471560 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚97413320 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450780 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26090 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177047040 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407872 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26088 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚393774412508672 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚440384725607936 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955591168 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1187 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚379833571209616 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007241 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 % _timestamp‚1.7885158657762454e+09 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 216192818 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚609 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917410816 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955591168 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 216218908 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚88891338 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1212 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411420 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚393773495097856 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚388394278834688 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚97413318 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚393773495097856 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410816 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5380115671040 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚611 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚434766571910144 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 118779500 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5380115671040 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚388394296837632 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26088 _step‚386 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚28315123 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5618177637888 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5379216263168 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚88890727 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5618406497792 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5359154471560 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚97413320 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450780 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26090 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177047040 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407872 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26088 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096‚ìöeR2Ú- _wandbruntime‚5492  _runtime‚5492ï)‘ÛøM \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5361160544506 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚388473927133184 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚88908809 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚609 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 398533632 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 216237307 SEDeviceStatsMonitor.on_train_batchÒ­sm"_end/active_bytes.small_pool.current‚ 899421184 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007241 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177047040 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚60588591 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5380329066496 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450988 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚379911551303744 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚97433216 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5381228487680 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26114 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5381228487680 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899421184 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 118830205 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31686 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚379911534264384 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚60588589 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚385272711848250 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5360263132878 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚97433218 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411628 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 _step‚387 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 118830205 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚388473927133184 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚28320220 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1187 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5380329066496 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚97433218 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚611 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚434855867190784 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5619580212736 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚385271797397262 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 216263423 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 118804091 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚440475471343616 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚440475235334144 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚393854238196736 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 118830205 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚388473927133184 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚28320220 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1187 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5380329066496 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚97433218 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚611 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚434855867190784 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5619580212736 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚385271797397262 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 216263423 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 118804091 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚440475471343616 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚440475235334144 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚393854238196736 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚88908809 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚609 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 398533632 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 216237307 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899421184 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007241 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177047040 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚60588591 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5380329066496 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450988 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚379911551303744 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚97433216 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5381228487680 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26114 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5381228487680 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899421184 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 118830205 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31686 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚379911534264384 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚60588589 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚385272711848250 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5360263132878 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚97433218 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411628 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 _step‚387 FDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚397810664455680 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 trainer/global_step‚4799 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚392375811615232 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407872 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚392375811615232 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚28603289 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚439232532946944 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚389148074456104 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411420 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚397811581866496 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214665 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 240217088 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5676644433920 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚28603907 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚439232509006848 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007241 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚444909153440768 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 _step‚390 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26088 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5435770251264 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5434870843392 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 120012402 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚61195015 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚61195013 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 120012402 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚392375793612288 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚620 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26090 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 120038490 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5435770251264 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚98408218 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.o³z]Än_train_batch_start/requested_bytes.all.current‚ 914450780 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1187 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5414601128668 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 398533632 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26088 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _runtime‚5545.717480671 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚98408220 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 398533632 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26088 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _runtime‚5545.717480671 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚98408220 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚397810664455680 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 trainer/global_step‚4799 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚392375811615232†» “å2ì- _wandbruntime‚5547  _runtime‚5547kzŒ'íúM N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚392455441910784 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚61207391 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚89817059 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚89816441 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 218465109 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26114 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 218491225 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚98428116 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5436883067904 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚397892324978688 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 120063107 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 240203776 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚439321852167680 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 216263680 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 120036993 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 _runtime‚5547.238104167 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚392455441910784 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899421184 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚98428116 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 120063107 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177047040 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚397891407554560 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26114 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚444999889121792 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 % _timestamp‚1.7885159217654457e+09 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚98428118 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚392455423907840 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 120036993 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007241 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚61207389 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31686 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 398533632 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411628 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450988 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚383810538970784 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚392455423907840 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899421184 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5435983646720 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 218491225 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚98428118 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚28609052 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚389226248760770·>–äìîÿM QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177047040 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚397891407554560 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26114 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚444999889121792 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 % _timestamp‚1.7885159217654457e+09 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚98428118 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚392455423907840 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 120036993 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007241 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚61207389 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31686 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 398533632 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411628 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450988 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚383810538970784 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚392455423907840 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899421184 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5435983646720 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 218491225 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚98428118 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚28609052 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚389226248760770 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚392455441910784 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚61207391 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚89817059 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚89816441 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 218465109 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26114 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 218491225 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚984281MŸÄ16 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5436883067904 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚397892324978688 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 120063107 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 240203776 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚439321852167680 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 216263680 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 120036993 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 _runtime‚5547.238104167 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚392455441910784 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899421184 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚98428116 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 120063107 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096‡Ï× ¯2ï- _wandbruntime‚5547  _runtime‚5547±>©[ðÕ 0loss/FRt3/assignment_loss‚2.4193057815233865 1loss/FBt3/assignment_loss‚0.10547632858321197 /loss/FRt4/detection_loss‚3.4443293389948932 1loss/SRqqt1/detection_loss‚1.8066427359978359 /loss/FBt4/detection_loss‚2.3728892554839454 2loss/SRqqt2/assignment_loss‚0.9321526047180999 0loss/FBt1/assignment_loss‚0.3883742426648193 /loss/FBt1/detection_loss‚2.3773478706677755 $loss/total_loss‚39.4584625713712 _runtime‚5547.239313833 % _timestamp‚1.7885159217690477e+09 0loss/FRt4/assignment_loss‚2.8003954508087854 /loss/FRt1/detection_loss‚2.0743246866126372 /loss/FRt3/detection_loss‚2.3886206104523606 .loss/FBt3/detection_loss‚2.604236743666909 .loss/FRt1/assignment_loss‚2.18628867822034 /loss/FRt2/detection_loss‚1.6568604288001856 0loss/FRt2/assignment_loss‚1.1336905817190805 1loss/SRqqt1/assignment_loss‚1.164271525417765 2loss/SRqqt3/assignment_loss‚1.2371610446799812 1loss/FBt2/assignment_loss‚0.13129055189619265 1loss/SRqqt2/detection_loss‚1.2462123526768252 /loss/FBt2/detection_loss‚0.6840717265828667 _step‚392 2loss/SRqqt4/assignment_loss‚1.9734066641330719 0loss/FBt4/assignment_loss‚0.5224425840885184 0loss/SRqqt3/detection_loss‚1.524931819377733 1loss/SRqqt4/detection_loss‚2.2837389636039735 epoch‚3 trainer/global_step‚4799vÙoT`ñÚ 2loss/SRqqt2/assignment_loss‚0.9321526047180999 0loss/FBt1/assignment_loss‚0.3883742426648193 /loss/FBt1/detection_loss‚2.3773478706677755 $loss/total_loss‚39.4584625713712 _runtime‚5547.239313833 % _timestamp‚1.7885159217690477e+09 0loss/FRt4/assignment_loss‚2.8003954508087854 /loss/FRt1/detection_loss‚2.0743246866126372 /loss/FRt3/detection_loss‚2.3886206104523606 .loss/FBt3/detection_loss‚2.604236743666909 .loss/FRt1/assignment_loss‚2.18628867822034 /loss/FRt2/detection_loss‚1.6568604288001856 0loss/FRt2/assignment_loss‚1.1336905817190805 1loss/SRqqt1/assignment_loss‚1.164271525417765 2loss/SRqqt3/assignment_loss‚1.2371610446799812 1loss/FBt2/assignment_loss‚0.13129055189619265 1loss/SRqqt2/detection_loss‚1.2462123526768252 /loss/FBt2/detection_loss‚0.6840717265828667 _step‚392 2loss/SRqqt4/assignment_loss‚1.9734066641330719 0loss/FBt4/assignment_loss‚0.5224425840885184 0loss/SRqqt3/detection_loss‚1.524931819377733 1loss/SRqqt4/detection_loss‚2.2837389636039735 epoch‚3 trainer/global_step‚4799 0loss/FRt3/assignment_loss‚2.4193057815233865 1loss/FBt3/assignment_loss‚0.10547632858321197 /loss/FRt4/detection_loss‚3.4443293389948932 1loss/SRqqt1/detection_loss‚1.8066427359978359 /loss/FBt4/detection_loss‚2.3728892554839454ˆ0 ¦=”ò‚(:‰ Ô¬êÔë›Çþ gpu.0.gpu‚38.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚171.501' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.87525 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0\Klysó‚(:è Ô¬êÔãûþ disk.sda1.out‚0 disk.sda2.in‚ 4.30078125! disk.sda2.out‚637.85693359375" disk.nvme1n1.out‚ 34784.8046875 cpu‚25.178800612422744 disk.sda.out‚637.85693359375) disk./.usagePercent‚75.85130293092904 disk.dm-0.out‚ 63979.6953125$ memory_percent‚96.00176811218262 disk.md0p1.in‚ 5.12890625! disk.nvme0n1.out‚ 29194.890625 disk.nvme1n1.in‚ 498.61328125 disk.sdb1.out‚0 disk.sdb2.in‚0.828125 disk.rbd2.in‚0 network.sent‚ 1.1515659e+07% disk./.usageGB‚2712.8099975585938 disk.sdb.out‚637.85693359375 disk.rbd0.out‚ 1.404296875# proc.memory.rssMB‚ 2990.52734375 disk.rbd2.out‚ 1.341796875! disk.sdb2.out‚637.85693359375) proc.memory.percent‚18.25273036956787 proc.cpu.threads‚17 disk.sda1.in‚0 disk.sdb.in‚0.828125 disk.sda.in‚ 4.30078125 disk.rbd0.in‚0 disk.md0.in‚ 5.12890625 network.recv‚ 1.804615e+06 disk.dm-0.in‚ 1102.05859375 disk.md0p1.out‚ 614.51171875 disk.nvme0n1.in‚ 603.4453125' proc.memory.availableMB‚ 655.0703125 disk.sdb1.in‚0 disk.md0.out‚ 614.51171875znúl“ô‚(:ˆ ã¬êÔ„ì´ÿ gpu.0.gpu‚14.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚118.142' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚29.5355 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0“䛕jõ‚(:ß ã¬êÔ÷冀 disk.rbd0.in‚0 disk.md0.out‚ 616.453125 network.recv‚ 1.810519e+06( disk./.usagePercent‚75.8502217131149 disk.sda1.in‚0 disk.dm-0.in‚ 1102.05859375 disk.sda.out‚639.85595703125 disk.sdb1.out‚0 disk.sdb2.in‚0.828125 disk.sda1.out‚0 disk.md0p1.in‚ 5.12890625 disk.nvme0n1.in‚ 603.4453125! disk.sda2.out‚639.85595703125$ memory_percent‚95.99876403808594 disk.dm-0.out‚ 64117.7890625 disk.md0p1.out‚ 616.453125 disk.sdb.out‚639.85595703125! disk.nvme1n1.out‚ 34822.390625 proc.cpu.threads‚17$ disk./.usageGB‚2712.771327972412 disk.rbd0.out‚ 1.404296875 disk.rbd2.in‚0! disk.sdb2.out‚639.85595703125# proc.memory.rssMB‚ 2990.52734375 cpu‚25.178400033973475 disk.sdb.in‚0.828125 disk.sdb1.in‚0 disk.rbd2.out‚ 1.341796875$ proc.memory.availableMB‚655.5625) proc.memory.percent‚18.25273036956787 network.sent‚ 1.1623318e+07" disk.nvme0n1.out‚ 29295.3984375 disk.sda2.in‚ 4.30078125 disk.nvme1n1.in‚ 498.61328125 disk.md0.in‚ 5.12890625 disk.sda.in‚ 4.30078125^/“-”ö‚(:‰ ò¬êÔ…‹þ gpu.0.gpu‚40.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚171.501' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.87525 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0&åÓp÷‚(:å ò¬êÔ«¦ÿ% disk./.usageGB‚2712.8322792053223# disk.nvme0n1.out‚29495.48828125 disk.rbd0.in‚0 disk.sdb2.in‚0.828125 cpu‚25.17801898919843 disk.dm-0.out‚ 64378.515625 network.sent‚ 1.1629771e+07) disk./.usagePercent‚75.85192593509716 disk.rbd2.in‚0 disk.sda1.in‚0 proc.cpu.threads‚17# disk.nvme1n1.out‚34883.02734375 disk.sdb.in‚0.828125 disk.sda2.in‚ 4.30078125 disk.sdb1.out‚0 disk.rbd2.out‚ 1.341796875 disk.sdb2.out‚641.2568359375' proc.memory.availableMB‚ 655.0703125 disk.rbd0.out‚ 1.404296875 disk.sdb1.in‚0 disk.md0.out‚ 617.80078125 disk.md0p1.out‚ 617.80078125$ memory_percent‚96.00176811218262# proc.memory.rssMB‚ 2990.52734375 disk.sda1.out‚0 disk.dm-0.in‚ 1102.05859375 disk.nvme0n1.in‚ 603.4453125 disk.sda.in‚ 4.30078125 disk.sda.out‚641.2568359375 disk.nvme1n1.in‚ 498.61328125 network.recv‚ 1.815663e+06 disk.sdb.out‚641.2568359375 disk.md0.in‚ 5.12890625 disk.md0p1.in‚ 5.12890625) proc.memory.percent‚18.25273036956787 disk.sda2.out‚641.2568359375¦EI“ø‚(:ˆ ­êÔξòý gpu.0.gpu‚2.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚37.0 gpu.0.powerWatts‚177.231' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚44.30775 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0§òú¾nù‚(:ã ­êÔßÈö€ disk.sda.in‚ 4.30078125# disk.nvme1n1.out‚34907.21484375 disk.md0.in‚ 5.12890625* proc.memory.percent‚18.307924270629883$ disk./.usageGB‚2712.957565307617 disk.md0p1.in‚ 5.12890625 disk.nvme1n1.in‚ 498.61328125 disk.sdb1.in‚0 disk.md0.out‚ 619.6328125$ memory_percent‚96.05517387390137 disk.sdb.in‚0.828125) disk./.usagePercent‚75.85542898695363 disk.dm-0.in‚ 1102.05859375 disk.dm-0.out‚64602.75! disk.sda2.out‚643.14794921875 disk.rbd0.in‚0 disk.rbd2.out‚ 1.341796875 cpu‚25.182154625034556 disk.sdb.out‚643.14794921875 disk.sda2.in‚ 4.30078125' proc.memory.availableMB‚ 646.3203125 network.recv‚ 1.819156e+06 disk.sda1.in‚0 disk.nvme0n1.in‚ 603.4453125 disk.rbd0.out‚ 1.404296875! disk.sdb2.out‚643.14794921875 proc.cpu.threads‚17 disk.sda1.out‚0# disk.nvme0n1.out‚29695.53515625 disk.sdb1.out‚0 disk.rbd2.in‚0" proc.memory.rssMB‚ 2999.5703125 network.sent‚ 1.1637294e+07 disk.md0p1.out‚ 619.6328125 disk.sda.out‚643.14794921875 disk.sdb2.in‚0.828125'[£’ú‚(:‡ ­êÔÑ«Âÿ gpu.0.gpu‚27.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚179.856' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚44.964 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0§õDeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 NHDeviceStatsMonitor.on_validation_batch_start/oversize_segments.allocated‚0 aMDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.freed‚389056556760768 LFDeviceStatsMonitor.on_validation_batch_start/active.large_pool.current‚2 VIDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.allocated‚91210211 VNDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.current‚205 D>DeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 OADeviceStatsMonitor.on_validation_batch_start/active.all.allocated‚ 221272329 UHDeviceStatsMonitor.on_validation_batch_start/active.large_pool.allocated‚99892970 [GDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.allocated‚403448031000576 SEDeviceStatsMonitor.on_validation_batch_start/allocation.all.allocated‚ 221272329 _KDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.freed‚450998327881728 OIDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.freed‚0 I?DeviceStatsMonitor.on_validation_batch_start/active.all.current‚20521 fRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.freed‚445254344302080 LFDeviceStatsMonitor.on_validation_batch_start/oversize_segments.current‚0 _PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 RDDeviceStatsMonitor.on_validation_batch_start/active.small_pool.freed‚ 121358840 UEDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.peak‚ 13955591168 cQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.allocated‚ 5500905936896 aTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.current‚23940096 bTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.current‚ 153258496 dRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.freed‚ 5743983579648 ZFDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.freed‚394536285383950 QIDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.allocated‚564 TKDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.peak‚1187 trainer/global_step‚4835 VHDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.freed‚ 121358840 OGDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.current‚207 hVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.allocated‚ 5744136838144 LDDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.peak‚607 WCDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.freed‚403447337881088 QKDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.allocated‚0 YIDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.allocated‚ 14220787712 cODeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.allocated‚450998505080320 ]NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.current‚ 1182793728 KBDeviceStatsMonitor.on_validation_batch_start/segment.all.allocated‚1171 J@DeviceStatsMonitor.on_validation_batch_start/allocation.all.peak‚31931 ZLDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.allocated‚ 121379359 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 ^JDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.allocated‚394536975568070 F=DeviceStatsMonitor.on_validation_batch_start/segment.all.peak‚1171 VHDeviceStatsMonitor.on_validation_batch_start/active.small_pool.allocated‚ 121379359 _MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.freed‚ 5500230820352 REDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.freed‚91210004 MDDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.peak‚1212 I@DeviceStatsMonitor.on_validation_batch_start/segment.all.current‚1171 _step‚393 bNDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.allocated‚397947125063680 ]ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.current‚ 673144760 ZKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 QIDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.allocated‚607 _MDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.freed‚ 5479728623182 K=DeviceStatsMonitor.on_validation_batch_start/active.all.freed‚ 221251808 ^JDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.allocated‚403448031000576 [LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.allocated‚29094724 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.freed‚29094519 OGDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.current‚564 PFDeviceStatsMonitor.on_validation_batch_start/active.small_pool.current‚20519 \JDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.freed‚ 5500230820352 eQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.allocated‚397947125063680 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 OGDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.current‚607 FDeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 NHDeviceStatsMonitor.on_validation_batch_start/oversize_segments.allocated‚0 aMDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.freed‚389056556760768 LFDeviceStatsMonitor.on_validation_batch_start/active.large_pool.current‚2 VIDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.allocated‚91210211 VNDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.current‚205 D>DeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 OADeviceStatsMonitor.on_validation_batch_start/active.all.allocated‚ 221272329 UHDeviceStatsMonitor.on_validation_batch_start/active.large_pool.allocated‚99892970 [GDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.allocated‚403448031000576 SEDeviceStatsMonitor.on_validation_batch_start/allocation.all.allocated‚ 221272329 _KDeviceStatsMonitor.on_validation_batch7· …{_start/inactive_split_bytes.all.freed‚450998327881728 OIDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.freed‚0 I?DeviceStatsMonitor.on_validation_batch_start/active.all.current‚20521 fRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.freed‚445254344302080 LFDeviceStatsMonitor.on_validation_batch_start/oversize_segments.current‚0 _PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 RDDeviceStatsMonitor.on_validation_batch_start/active.small_pool.freed‚ 121358840 UEDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.peak‚ 13955591168 cQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.allocated‚ 5500905936896 aTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.current‚23940096 bTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.current‚ 153258496 dRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.freed‚ 5743983579648 ZFDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.freed‚394536285383950 QIDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.allocated‚564 TKDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.peak‚1187 trainer/global_step‚4835 VHDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.freed‚ 121358840 OGDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.current‚207 hVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.allocated‚ 5744136838144 LDDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.peak‚607 WCDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.freed‚403447337881088 QKDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.allocated‚0 YIDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.allocated‚ 14220787712 cODeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.allocated‚450998505080320 ]NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.current‚ 1182793728 KBDeviceStatsMonitor.on_validation_batch_start/segment.all.allocated‚1171 J@DeviceStatsMonitor.on_validation_batch_start/allocation.all.peak‚31931 ZLDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.allocated‚ 121379359 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 ^JDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.allocated‚394536975568070 F=DeviceStatsMonitor.on_validation_batch_start/segment.all.peak‚1171 VHDeviceStatsMonitor.on_validation_batch_start/active.small_pool.allocated‚ 121379359 _MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.freed‚ 5500230820352 REDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.freed‚91210004 MDDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.peak‚1212 I@DeviceStatsMonitor.on_validation_batch_start/segment.all.current‚1171 _step‚393 bNDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.allocated‚397947125063680 ]ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.current‚ 673144760 ZKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 QIDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.allocated‚607 _MDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.freed‚ 5479728623182 K=DeviceStatsMonitor.on_validation_batch_start/active.all.freed‚ 221251808 ^JDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.allocated‚403448031000576 [LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.allocated‚29094724 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.freed‚29094519 OGDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.current‚564 PFDeviceStatsMonitor.on_validation_batch_start/active.small_pool.current‚20519 \JDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.freed‚ 5500230820352 eQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.allocated‚397947125063680‰g*S‹2ƒ- _wandbruntime‚5644  _runtime‚5644ÈŸŠf)„àR KBDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.peak‚1212 OIDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.allocated‚0 OBDeviceStatsMonitor.on_validation_batch_end/active.large_pool.freed‚99908244 [MDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.current‚ 673144760 OEDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.peak‚31686 QCDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.current‚ 693119488 UADeviceStatsMonitor.on_validation_batch_end/active_bytes.all.freed‚403501397719552 TFDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.current‚ 690184120 _KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.freed‚398000678336512 JDDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.peak‚0 \HDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.allocated‚403502090839040 YJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.peak‚ 1177047040 RHDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.current‚20519 TFDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.freed‚ 121367614 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.allocated‚62124927 HBDeviceStatsMonitor.on_validation_batch_end/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.peak‚607 hTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.allocated‚445309817714176 C=DeviceStatsMonitor.on_validation_batch_end/num_oom_rejections‚0 RBDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.peak‚ 14220787712 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.freed‚0 BDeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚31931 YKDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.current‚ 177198592 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 ZJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 _RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.current‚23940096 @9DeviceStatsMonitor.on_validation_batch_end/max_split_size‚-1 YJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.peak‚ 1175007241 ICDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.freed‚0 ]KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.freed‚ 5500719383040 PIDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.peak‚35 UEDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.current‚ 14220787712 I@DeviceStatsMonitor.on_validation_batch_end/segment.all.allocated‚1171 NEDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.peak‚3677 RLDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.current‚2 E?DeviceStatsMonitor.on_validation_batch_end/num_sync_all_streams‚2 ICDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.freed‚0 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.freed‚0 ZHDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.freed‚ 5500719383040 aMDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.allocated‚451054443115008 \LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.current‚ 13037993984 ]NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 JDDeviceStatsMonitor.on_validation_batch_end/active.large_pool.current‚2 NDDeviceStatsMonitor.on_validation_batch_end/active.small_pool.current‚20519 JDDeviceStatsMonitor.on_validation_batch_end/oversize_segments.current‚0 XDDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.freed‚394586542400498 [LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.current‚ 1182793728 I;DeviceStatsMonitor.on_validation_batch_end/active.all.freed‚ 221275858 dPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.freed‚445309793774080 93DeviceStatsMonitor.on_validation_batch_end/num_ooms‚0 ICDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.freed‚0 \HDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.freed‚398000678336512 [MDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.current‚ 675116544 XJDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.allocated‚ 121388133 BDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚1171 KADeviceStatsMonitor.on_validation_batch_end/active.small_pool.peak‚31686 NHDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.current‚2 VHDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.peak‚ 398533632 MEDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.current‚564 $ _timestamp‚1.788516018893131e+09 `LDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.allocated‚398000696339456 bPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.freed‚ 5744472142336 LFDeviceStatsMonitor.on_validation_batch_end/oversize_segments.allocated‚0 ZMDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.current‚17039360 OGDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.allocated‚564 SFDeviceStatsMonitor.on_validation_batch_end/active.large_pool.allocated‚99908246 VGDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.peak‚ 1177047040 SCDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.peak‚ 13955591168 KADeviceStatsMonitor.on_validation_batch_end/allocation.all.current‚20521 M?DeviceStatsMonitor.on_validation_batch_end/allocation.all.freed‚ 221275858 TLDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.current‚205 @:DeviceStatsMonitor.on_validation_batch_end/num_device_free‚0 _KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.freed‚389106326850592 M?DeviceStatsMonitor.on_validation_batch_end/active.all.allocated‚ 221296379 PBDeviceStatsMonitor.on_validation_batch_end/active.small_pool.freed‚ 121367614 cODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.allocated‚398000696339456 GADeviceStatsMonitor.on_validation_batch_end/oversize_segments.peak‚0 OGDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.allocated‚607 MEDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.current‚607 trainer/global_step‚4835 ZMDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.current‚18002944 aODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.allocated‚ 5501394499584 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 XIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 JBDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.peak‚564 _step‚394 QCDeviceStatsMonitor.on_validation_batch_end/allocation.all.allocated‚ 221296379 ^NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 TGDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.allocated‚91225312 P@DeviceStatsMonitor.on_validation_batch_end/active_bytes.all.peak‚ 13955591168 XJDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.current‚ 675116544 `RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.current‚ 153258496 cODeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.allocated‚389106343889952dE‹k)…åR QCDeviceStatsMonitor.on_validation_batch_end/allocation.all.allocated‚ 221296379 ^NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 TGDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.allocated‚91225312 P@DeviceStatsMonitor.on_validation_batch_end/active_bytes.all.peak‚ 13955591168 XJDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.current‚ 675116544 `RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.current‚ 153258496 cODeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.allocated‚389106343889952 KBDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.peak‚1212 OIDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.allocated‚0 OBDeviceStatsMonitor.on_validation_batch_end/active.large_pool.freed‚99908244 [MDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.current‚ 673144760 OEDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.peak‚31686 QCDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.current‚ 693119488 UADeviceStatsMonitor.on_validation_batch_end/active_bytes.all.freed‚403501397719552 TFDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.current‚ 690184120 _KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.freed‚398000678336512 JDDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.peak‚0 \HDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.allocated‚403502090839040 YJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.peak‚ 1177047040 RHDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.current‚20519 TFDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.freed‚ 121367614 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.allocated‚62124927 HBDeviceStatsMonitor.on_validation_batch_end/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.peak‚607 hTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.allocated‚445309817714176 C=DeviceStatsMonitor.on_validation_batch_end/num_oom_rejections‚0 RBDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.peak‚ 14220787712 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.freed‚0 BDeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚31931 YKDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.current‚ 177198592 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 ZJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 _RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.current‚23940096 @9DeviceStatsMonitor.on_validation_batch_end/max_split_size‚-1 YJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.peak‚ 1175007241 ICDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.freed‚0 ]KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.freed‚ 5500719383040 PIDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.peak‚35 UEDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.current‚ 14220787712 I@DeviceStatsMonitor.on_validation_batch_end/segment.all.allocated‚1171 NEDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.peak‚3677 RLDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.current‚2 E?DeviceStatsMonitor.on_validation_batch_end/num_sync_all_streams‚2 ICDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.freed‚0 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.freed‚0 ZHDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.freed‚ 5500719383040 aMDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.allocated‚451054443115008 \LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.current‚ 13037993984 ]NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 JDDeviceStatsMonitor.on_validation_batch_end/active.large_pool.current‚2 NDDeviceStatsMonitor.on_validation_batch_end/active.small_pool.current‚20519 JDDeviceStatsMonitor.on_validation_batch_end/oversize_segments.current‚0 XDDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.freed‚394586542400498 [LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.current‚ 1182793728 I;DeviceStatsMonitor.on_validation_batch_end/active.all.freed‚ 221275858 dPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.freed‚445309793774080 93DeviceStatsMonitor.on_validation_batch_end/num_ooms‚0 ICDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.freed‚0 \HDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.freed‚398000678336512 [MDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.current‚ 675116544 XJDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.allocated‚ 121388133 BDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚1171 KADeviceStatsMonitor.on_validation_batch_end/active.small_pool.peak‚31686 NHDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.current‚2 VHDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.peak‚ 398533632 MEDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.current‚564 $ _timestamp‚1.788516018893131e+09 `LDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.allocated‚398000696339456 bPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.freed‚ 5744472142336 LFDeviceStatsMonitor.on_validation_batch_end/oversize_segments.allocated‚0 ZMDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.current‚17039360 OGDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.allocated‚564 SFDeviceStatsMonitor.on_validation_batch_end/active.large_pool.allocated‚99908246 VGDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.peak‚ 1177047040 SCDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.peak‚ 13955591168 KADeviceStatsMonitor.on_validation_batch_end/allocation.all.current‚20521 M?DeviceStatsMonitor.on_validation_batch_end/allocation.all.freed‚ 221275858 TLDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.current‚205 @:DeviceStatsMonitor.on_validation_batch_end/num_device_free‚0 _KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.freed‚389106326850592 M?DeviceStatsMonitor.on_validation_batch_end/active.all.allocated‚ 221296379 PBDeviceStatsMonitor.on_validation_batch_end/active.small_pool.freed‚ 121367614 cODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.allocated‚398000696339456 GADeviceStatsMonitor.on_validation_batch_end/oversize_segments.peak‚0 OGDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.allocated‚607 MEDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.current‚607 trainer/global_step‚4835 ZMDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.current‚18002944 aODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.allocated‚ 5501394499584 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 XIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 JBDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.peak‚564 _step‚394Š”kí£”†‚(:‰ ½­êÔ߉œÿ gpu.0.gpu‚60.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚36.0 gpu.0.powerWatts‚114.131' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.53275 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0jGÀ]s‡‚(:è ½­êÔªÜÓ€$ memory_percent‚96.05624675750732 disk.rbd2.out‚ 1.341796875# disk./.usageGB‚2712.52693939209 disk.rbd0.in‚0 disk.sdb1.in‚0* proc.memory.percent‚18.307924270629883 network.recv‚ 1.835988e+06 disk.sda1.in‚0# disk.nvme0n1.out‚30092.15234375 disk.sda2.in‚ 4.30078125 disk.nvme1n1.in‚ 498.61328125 disk.sdb.in‚0.828125 disk.sdb.out‚651.53466796875 disk.rbd0.out‚ 1.404296875! disk.sdb2.out‚651.53466796875" proc.memory.rssMB‚ 2999.5703125 disk.sda1.out‚0! disk.sda2.out‚651.53466796875 disk.md0.in‚ 5.12890625 cpu‚25.226532580418958 proc.cpu.threads‚17 disk.sdb1.out‚0 disk.dm-0.in‚ 1102.05859375 disk.dm-0.out‚ 65377.53125 disk.md0p1.in‚ 5.12890625 disk.sda.in‚ 4.30078125 disk.sda.out‚651.53466796875# disk.nvme1n1.out‚35285.37890625( proc.memory.availableMB‚ 646.14453125 disk.md0.out‚ 627.73828125) disk./.usagePercent‚75.84338850612453 disk.nvme0n1.in‚ 603.4453125 disk.rbd2.in‚0 network.sent‚ 1.1671624e+07 disk.md0p1.out‚ 627.73828125 disk.sdb2.in‚0.828125ól<2ˆ- _wandbruntime‚5658  _runtime‚5658D¾Á`‰¾§ =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.3124999701976776 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.3194444477558136 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.8571428656578064 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 jet/accuracy_7_of_7‚0 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5206758975982666 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.24655170738697052 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.4854591190814972 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.4829787611961365 :"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.39559388160705566 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.2602807581424713 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.07630948722362518 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 *"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.5 6 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.987833559513092 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.5489911437034607 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 :"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.47629886865615845 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.7727272510528564 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9184028506278992 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 ( Purity/1FRt*SRqqt4FBt/FRt_purity‚0.5 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 6 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.994355320930481 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.1995239555835724 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.4195736348628998 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.20120622217655182 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 & Purity/1FRt2SRqqt2FBt/FBt_purity‚1 jet/accuracy_2_of_7‚0 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9787499904632568 6 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.529411792755127 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.02351347729563713 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6396048069000244 :"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.43829837441444397 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.7575758099555969 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.4341980218887329 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 7 Purity/*FRt3SRqqt*FBt/FBt_purity‚0.9285714030265808 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 particle/accuracy_6_of_6‚1 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9933333396911621 & Purity/1FRt3SRqqt2FBt/FRt_purity‚1 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 particle/accuracy_5_of_5‚1 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.9444444179534912 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8182422518730164 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9937499761581421 8"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.557539701461792 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 9"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.4094771444797516 & Purity/1FRt0SRqqt4FBt/FRt_purity‚1 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9907218813896179 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9966723322868347 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5491482615470886 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 particle/accuracy_1_of_1‚1 particle/accuracy_1_of_3‚1 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 8 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.49881696701049805 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 particle/accuracy_2_of_2‚1 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.09441624581813812 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 9"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.3829965889453888 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.20811952650547028 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 <"Purity/1FRt2SR}1´~qqt0FBt/event_purity‚0.0025012025143951178 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.24142533540725708 7 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.3484848737716675 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.4561403691768646 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 ( Purity/3FRt4SRqqt*FBt/FRt_purity‚0.5 & Purity/*FRt3SRqqt2FBt/FRt_purity‚1 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 :"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.45561301708221436 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 particle/accuracy_4_of_5‚1 /"Purity/0FRt0SRqqt3FBt/event_purity‚0.203125 :"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.47239184379577637 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.4097222089767456 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.17694681882858276 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.17210423946380615 :"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.45312222838401794 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 7 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.2155555784702301 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.4109366536140442 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.5315476059913635 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 ;"Purity/0FRt2SRqqt*FBt/event_purity‚0.055020369589328766 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 particle/accuracy_4_of_7‚1 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 +"Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0.75 7 Purity/2FRt*SRqqt1FBt/FBt_purity‚0.9960317611694336 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 jet/accuracy_4_of_6‚0 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.2291666716337204 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.2055044174194336 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.21555742621421814 *jet/accuracy_1_of_1‚0.4689030349254608 8 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.31018516421318054 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.2574805021286011 :"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.45353561639785767 :"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.43848684430122375 :"Purity/0FRt1SRqqt2FBt/event_purity‚0.08477012068033218 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.03240030258893967 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.4837905168533325 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.23863635957241058 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 7 Purity/*FRt3SRqqt1FBt/FBt_purity‚0.9259259104728699 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 particle/accuracy_1_of_7‚1 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9078014492988586 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9827999472618103 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 -jet/accuracy_4_of_4‚0.0025489861145615578 ."Purity/3FRt0SRqqt1FBt/event_purity‚0.03125 (particle/accuracy‚0.8535776734352112 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 jet/accuracy_5_of_7‚0 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9979472756385803 7 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.2142140120267868 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.20909996330738068 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.5104166865348816 -"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.4375 -"Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.3125 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 ) Purity/4FRt1SRqqt0FBt/FRt_purity‚0.25 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9776960611343384 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 *"Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0.5 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9701754450798035 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.53125 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 & Purity/1FRt4SRqqt0FBt/FRt_purity‚0 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.27375444769859314 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9980158805847168 6 Purity/2FRt2SRqqt1FBt/FBt_purity‚0.976190447807312 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.39945492148399353 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.19857029616832733 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9926417469978333 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 +jet/accuracy_2_of_2‚0.10626085102558136 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.20000000298023224 jet/accuracy_1_of_6‚1 5 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.80952388048172 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.4487099051475525 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 'particle/f_score‚0.4171363115310669 7 Purity/0FRt3SRqqt*FBt/FBt_purity‚0.7777777910232544 & Purity/*FRt0SRqqt4FBt/FRt_purity‚1 :"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.43289849162101746 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9919438362121582 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.5440270900726318 :"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.42082294821739197 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.5277777910232544 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 7 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.1810699701309204 :"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.46666666865348816 7 Purity/3FRt1SRqqt2FBt/FRt_purity‚0.3333333432674408 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 particle/accuracy_3_of_6‚1 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 % _timestamp‚1.7885160333758934e+09 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 jet/accuracy_4_of_7‚0 particle/accuracy_7_of_7‚1 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚1 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.5266978740692139 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 <"Purity/2FRt1SRqqt*FBt/event_purity‚0.0014430014416575432 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.17424243688583374 ,"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.125 * Purity/4FRt*SRqqt*FBt/FRt_purity‚0.125 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.24436679482460022 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.10717656463384628 ("Purity/1FRt0SRqqt3FBt/event_purity‚0 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 + Purity/0FRt*SRqqt4FBt/FBt_purity‚0.8125 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 particle/accuracy_2_of_7‚1 :"Purity/0FRt*SRqqt0FBt/event_purity‚0.35824429988861084 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.11224953085184097 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 particle/accuracy_2_of_4‚1 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 particle/accuracy_4_of_6‚1 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 jet/accuracy_5_of_5‚0 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 epoch‚3 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9959936738014221 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.48244521021842957 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.4978581964969635 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.5344516634941101 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 :"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.43879416584968567 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9866386651992798 9"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.3807455003261566 particle/accuracy_3_of_4‚1 particle/accuracy_5_of_6‚1 :"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.40838783979415894 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 particle/accuracy_3_of_7‚1 ) Purity/*FRt1SRqqt3FBt/FRt_purity‚0.25 *"Purity/4FRt2SRqqt*FBt/SRqqt_purity‚0.5 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.3537655174732208 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.15517443418502808 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.01315429899841547 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.4517800509929657 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 9"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.4466666877269745 ,jet/accuracy_3_of_6‚0.057692307978868484 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.22543857991695404 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 ("Purity/0FRt3SRqqt*FBt/event_purity‚0 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.35868051648139954 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.22328144311904907 +jet/accuracy_2_of_4‚0.35047098994255066 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 *loss/val_total_loss‚41.447875907265775 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.4464285671710968 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 9"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.4409805238246918 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8500000238418579 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 ,jet/accuracy_3_of_4‚0.050572413951158524 ("Purity/*FRt3SRqqt0FBt/event_purity‚0 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.23677696287631989 +particle/specificity‚0.9494189023971558 ("Purity/0FRt3SRqqt0FBt/event_purity‚0 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.36092764139175415 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.4638157784938812 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.22697456181049347 particle/accuracy_1_of_5‚1 7 Purity/0FRt3SRqqt1FBt/FBt_purity‚0.7777777910232544 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.3688271939754486 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9989040493965149 ("Purity/1FRt*SRqqt3FBt/event_purity‚0 7 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.2850574553012848 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.007979975081980228 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.23060110211372375 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9594109058380127 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.10794727504253387 ;"Purity/2FRt*SRqqt2FBt/event_purity‚0.011627906933426857 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 particle/accuracy_3_of_5‚1 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.21735113859176636 7 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.3090900778770447 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 +"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.25 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 ) Purity/1FRt1SRqqt3FBt/FRt_purity‚0.25 7 Purity/0FRt2SRqqt1FBt/FBt_purity‚0.9773584604263306 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.43089085817337036 ,"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.125 :"Purity/0FRt*SRqqt2FBt/event_purity‚0.28312182426452637 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9897712469100952 9"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.3909356892108917 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.020591402426362038 :"Purity/2FRt0SRqqt2FBt/event_purity‚0.02631578966975212 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 7 Purity/2FRt2SRqqt*FBt/FBt_purity‚0.9767441749572754 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 +jet/accuracy_3_of_5‚0.10910136997699738 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.5415292978286743 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 jet/accuracy_5_of_6‚0 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8518518209457397 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 :"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.49537035822868347 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.3993056118488312 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 *jet/accuracy_1_of_5‚0.9728110432624817 particle/accuracy_1_of_2‚1 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.1269841343164444 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.23861828446388245 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.5124558806419373 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.3333333432674408 ("Purity/*FRt2SRqqt3FBt/SRqqt_purity‚1 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.4987383484840393 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.2901221513748169 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.05624999850988388 ;"Purity/2FRt1SRqqt1FBt/event_purity‚0.004464285913854837 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 *jet/accuracy_1_of_4‚0.9165971875190735 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.4863376319408417 ;"Purity/0FRt2SRqqt1FBt/event_purity‚0.028301887214183807 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 <"Purity/3FRt*SRqqt*FBt/event_purity‚0.0023419205099344254 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 particle/accuracy_1_of_4‚1 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.14473684132099152 + Purity/0FRt0SRqqt4FBt/FBt_purity‚0.8125 particle/accuracy_5_of_7‚1 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 ( Purity/3FRt4SRqqt0FBt/FRt_purity‚0.5 ( Purity/*FRt*SRqqt4FBt/FRt_purity‚0.5 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 ,"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.875 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9846230149269104 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.22936512529850006 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.20709876716136932 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 +jet/accuracy_2_of_5‚0.47870585322380066 *"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.5 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9884464740753174 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 jet/accuracy_6_of_6‚0 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5687980055809021 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9936458468437195 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 :"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.30000001192092896 *jet/accuracy_2_of_6‚0.4487179219722748 7 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.2380952537059784 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.01965399459004402 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 :"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.40740740299224854 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 7 Purity/0FRt2SRqqt2FBt/FBt_purity‚0.9821428656578064 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 ("Purity/0FRt2SRqqt3FBt/SRqqt_purity‚1 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.17810943722724915 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.2576174736022949 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9753472208976746 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.19649378955364227 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.20998536050319672 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.3491070866584778 9"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.4166666567325592 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5567396879196167 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 7 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.2454955130815506 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 7 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.3166666626930237 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9830560088157654 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 + Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9375 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 *jet/accuracy_1_of_3‚0.8635676503181458 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 9"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.3884044289588928 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.4516921043395996 :"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.47855308651924133 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚1 trainer/global_step‚4835 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.4512235224246979 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9351851940155029 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.4162713587284088 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.19811655580997467 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2941986620426178 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 particle/accuracy_4_of_4‚1 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.3194444477558136 ) Purity/4FRt1SRqqt*FBt/FRt_purity‚0.25 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 :"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.43209874629974365 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 ("Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.25925925374031067 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 ;"Purity/3FRt*SRqqt1FBt/event_purity‚0.008333333767950535 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.2222222238779068 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 *jet/accuracy_1_of_2‚0.7377653121948242 7 Purity/*FRt*SRqqt4FBt/FBt_purity‚0.8333333134651184 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.4166666567325592 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.23530003428459167 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 *"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.5 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.4899425506591797 :"Purity/*FRt0SRqqt2FBt/event_purity‚0.26025593280792236 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9706541895866394 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.3697487711906433 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 ;"Purity/1FRt1SRqqt1FBt/event_purity‚0.047168973833322525 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.4322513937950134 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 8"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.574305534362793 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.36307236552238464 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 +jet/accuracy_2_of_3‚0.24079462885856628 particle/accuracy_3_of_3‚1 + Purity/1FRt3SRqqt1FBt/FRt_purity‚0.3125 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.2291666716337204 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 ;"Purity/1FRt1SRqqt2FBt/event_purity‚0.004166666883975267 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 particle/accuracy_2_of_3‚1 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 8 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.25648146867752075 :"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.30000001192092896 , Purity/*FRt2SRqqt2FBt/FRt_purity‚0.28125 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 jet/accuracy_6_of_7‚0 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.20909996330738068 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 :"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.44999998807907104 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.2857142984867096 jet/accuracy_1_of_7‚1 9"Purity/*FRt2SRqqt0FBt/event_purity‚0.0256629791110754 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 8"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.511054515838623 +particle/sensitivity‚0.3357149660587311 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.4285714328289032 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 ("Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0 _runtime‚5658.849324032 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.4538044333457947 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.09219858795404434 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.4183308184146881 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.4545099139213562 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 8 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.37291669845581055 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.3150291442871094 ("Purity/*FRt3SRqqt*FBt/event_purity‚0 :"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.44533976912498474 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.4465302526950836 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9864341616630554 ,jet/accuracy_3_of_3‚0.027545208111405373 particle/accuracy_2_of_6‚1 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9659090638160706 7 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.2666666805744171 & Purity/1FRt4SRqqt*FBt/FRt_purity‚0 8"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.949999988079071 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.3065249025821686 ;"Purity/1FRt2SRqqt1FBt/event_purity‚0.008620689623057842 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2738095223903656 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 jet/accuracy_3_of_7‚0 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9878169894218445 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.21987681090831757 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.20000000298023224 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 +"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.25 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9868918061256409 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.4346936047077179 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.8874269723892212 particle/accuracy_6_of_7‚1 * Purity/4FRt*SRqqt0FBt/FRt_purity‚0.125 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.1965513676404953 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 :"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.46128135919570923 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 ;"Purity/1FRt2SRqqt*FBt/event_purity‚0.002801120514050126 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9976190328598022 9"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.3768768012523651 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9966723322868347 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.06284764409065247 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.01367811020463705 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.19348660111427307 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 +"Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0.75 _step‚395 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.5263158082962036 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9820101857185364 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.4465302526950836 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9219577312469482 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9843736290931702 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.24029658734798431 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 ;"Purity/3FRt0SRqqt*FBt/event_purity‚0.011111111380159855 ) Purity/2FRt3SRqqt1FBt/FRt_purity‚0.25 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9963061213493347 :"Purity/*FRt1SRqqt2FBt/event_purity‚0.04203788936138153 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2587589621543884 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 *"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.5 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9743308424949646 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.3653067946434021 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9965516924858093 7 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.2081461250782013 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6893441081047058 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.29467183351516724 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9773243069648743 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.4217425286769867 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.4527410864830017 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 ,jet/accuracy_4_of_5‚0.007392473053187132 6validation_average_jet_accuracy‚0.4254087209701538 particle/accuracy_1_of_6‚1 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 particle/accuracy_2_of_5‚1 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.014040060341358185 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.030083145946264267 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 *"Purity/4FRt2SRqqt0FBt/SRqqt_purity‚0.5 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812*’Ò>Šç ;"Purity/0FRt2SRqqt*FBt/event_purity‚0.055020369589328766 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 particle/accuracy_4_of_7‚1 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 +"Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0.75 7 Purity/2FRt*ù8¸ûùSRqqt1FBt/FBt_purity‚0.9960317611694336 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 jet/accuracy_4_of_6‚0 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.2291666716337204 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.2055044174194336 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.21555742621421814 *jet/accuracy_1_of_1‚0.4689030349254608 8 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.31018516421318054 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.2574805021286011 :"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.45353561639785767 :"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.43848684430122375 :"Purity/0FRt1SRqqt2FBt/event_purity‚0.08477012068033218 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.03240030258893967 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.4837905168533325 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.23863635957241058 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 7 Purity/*FRt3SRqqt1FBt/FBt_purity‚0.9259259104728699 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 particle/accuracy_1_of_7‚1 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9078014492988586 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9827999472618103 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 -jet/accuracy_4_of_4‚0.0025489861145615578 ."Purity/3FRt0SRqqt1FBt/event_purity‚0.03125 (particle/accuracy‚0.8535776734352112 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 jet/accuracy_5_of_7‚0 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9979472756385803 7 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.2142140120267868 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.20909996330738068 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.5104166865348816 -"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.4375 -"Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.3125 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 ) Purity/4FRt1SRqqt0FBt/FRt_purity‚0.25 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9776960611343384 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 *"Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0.5 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9701754450798035 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.53125 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 & Purity/1FRt4SRqqt0FBt/FRt_purity‚0 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.27375444769859314 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9980158805847168 6 Purity/2FRt2SRqqt1FBt/FBt_purity‚0.976190447807312 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.39945492148399353 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.19857029616832733 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9926417469978333 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 +jet/accuracy_2_of_2‚0.10626085102558136 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.20000000298023224 jet/accuracy_1_of_6‚1 5 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.80952388048172 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.4487099051475525 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 'particle/f_score‚0.4171363115310669 7 Purity/0FRt3SRqqt*FBt/FBt_purity‚0.7777777910232544 & Purity/*FRt0SRqqt4FBt/FRt_purity‚1 :"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.43289849162101746 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9919438362121582 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.5440270900726318 :"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.42082294821739197 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.5277777910232544 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 7 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.1810699701309204 :"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.46666666865348816 7 Purity/3FRt1SRqqt2FBt/FRt_purity‚0.3333333432674408 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 particle/accuracy_3_of_6‚1 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 % _timestamp‚1.7885160333758934e+09 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 jet/accuracy_4_of_7‚0 particle/accuracy_7_of_7‚1 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚1 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.5266978740692139 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 <"Purity/2FRt1SRqqt*FBt/event_purity‚0.0014430014416575432 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.17424243688583374 ,"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.125 * Purity/4FRt*SRqqt*FBt/FRt_purity‚0.125 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.24436679482460022 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.10717656463384628 ("Purity/1FRt0SRqqt3FBt/event_purity‚0 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 + Purity/0FRt*SRqqt4FBt/FBt_purity‚0.8125 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 particle/accuracy_2_of_7‚1 :"Purity/0FRt*SRqqt0FBt/event_purity‚0.35824429988861084 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.11224953085184097 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 particle/accuracy_2_of_4‚1 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 particle/accuracy_4_of_6‚1 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 jet/accuracy_5_of_5‚0 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 epoch‚3 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9959936738014221 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.48244521021842957 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.4978581964969635 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.5344516634941101 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 :"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.43879416584968567 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9866386651992798 9"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.3807455003261566 particle/accuracy_3_of_4‚1 particle/accuracy_5_of_6‚1 :"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.40838783979415894 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 particle/accuracy_3_of_7‚1 ) Purity/*FRt1SRqqt3FBt/FRt_purity‚0.25 *"Purity/4FRt2SRqqt*FBt/SRqqt_purity‚0.5 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.3537655174732208 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.15517443418502808 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.01315429899841547 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.4517800509929657 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 9"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.4466666877269745 ,jet/accuracy_3_of_6‚0.057692307978868484 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.22543857991695404 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 ("Purity/0FRt3SRqqt*FBt/event_purity‚0 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.35868051648139954 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.22328144311904907 +jet/accuracy_2_of_4‚0.35047098994255066 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 *loss/val_total_loss‚41.447875907265775 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.4464285671710968 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 9"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.4409805238246918 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8500000238418579 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 ,jet/accuracy_3_of_4‚0.050572413951158524 ("Purity/*FRt3SRqqt0FBt/event_purity‚0 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.23677696287631989 +particle/specificity‚0.9494189023971558 ("Purity/0FRt3SRqqt0FBt/event_purity‚0 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.36092764139175415 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.4638157784938812 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.22697456181049347 particle/accuracy_1_of_5‚1 7 Purity/0FRt3SRqqt1FBt/FBt_purity‚0.7777777910232544 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.3688271939754486 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9989040493965149 ("Purity/1FRt*SRqqt3FBt/event_purity‚0 7 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.2850574553012848 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.007979975081980228 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.23060110211372375 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9594109058380127 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.10794727504253387 ;"Purity/2FRt*SRqqt2FBt/event_purity‚0.011627906933426857 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 particle/accuracy_3_of_5‚1 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.21735113859176636 7 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.3090900778770447 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 +"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.25 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 ) Purity/1FRt1SRqqt3FBt/FRt_purity‚0.25 7 Purity/0FRt2SRqqt1FBt/FBt_purity‚0.9773584604263306 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.43089085817337036 ,"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.125 :"Purity/0FRt*SRqqt2FBt/event_purity‚0.28312182426452637 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9897712469100952 9"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.3909356892108917 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.020591402426362038 :"Purity/2FRt0SRqqt2FBt/event_purity‚0.02631578966975212 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 7 Purity/2FRt2SRqqt*FBt/FBt_purity‚0.9767441749572754 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 +jet/accuracy_3_of_5‚0.10910136997699738 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.5415292978286743 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 jet/accuracy_5_of_6‚0 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8518518209457397 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 :"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.49537035822868347 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.3993056118488312 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 *jet/accuracy_1_of_5‚0.9728110432624817 particle/accuracy_1_of_2‚1 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.1269841343164444 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.23861828446388245 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.5124558806419373 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.3333333432674408 ("Purity/*FRt2SRqqt3FBt/SRqqt_purity‚1 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.4987383484840393 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.2901221513748169 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.05624999850988388 ;"Purity/2FRt1SRqqt1FBt/event_purity‚0.004464285913854837 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 *jet/accuracy_1_of_4‚0.9165971875190735 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.4863376319408417 ;"Purity/0FRt2SRqqt1FBt/event_purity‚0.028301887214183807 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 <"Purity/3FRt*SRqqt*FBt/event_purity‚0.0023419205099344254 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 particle/accuracy_1_of_4‚1 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.14473684132099152 + Purity/0FRt0SRqqt4FBt/FBt_purity‚0.8125 particle/accuracy_5_of_7‚1 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 ( Purity/3FRt4SRqqt0FBt/FRt_purity‚0.5 ( Purity/*FRt*SRqqt4FBt/FRt_purity‚0.5 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 ,"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.875 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9846230149269104 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.22936512529850006 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.20709876716136932 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 +jet/accuracy_2_of_5‚0.47870585322380066 *"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.5 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9884464740753174 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 jet/accuracy_6_of_6‚0 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5687980055809021 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9936458468437195 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 :"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.30000001192092896 *jet/accuracy_2_of_6‚0.4487179219722748 7 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.2380952537059784 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.01965399459004402 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 :"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.40740740299224854 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 7 Purity/0FRt2SRqqt2FBt/FBt_purity‚0.9821428656578064 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 ("Purity/0FRt2SRqqt3FBt/SRqqt_purity‚1 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.17810943722724915 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.2576174736022949 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9753472208976746 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.19649378955364227 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.20998536050319672 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.3491070866584778 9"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.4166666567325592 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5567396879196167 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 7 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.2454955130815506 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 7 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.3166666626930237 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9830560088157654 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 + Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9375 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 *jet/accuracy_1_of_3‚0.8635676503181458 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 9"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.3884044289588928 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.4516921043395996 :"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.47855308651924133 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚1 trainer/global_step‚4835 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.4512235224246979 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9351851940155029 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.4162713587284088 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.19811655580997467 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2941986620426178 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 particle/accuracy_4_of_4‚1 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.3194444477558136 ) Purity/4FRt1SRqqt*FBt/FRt_purity‚0.25 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 :"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.43209874629974365 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 ("Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.25925925374031067 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 ;"Purity/3FRt*SRqqt1FBt/event_purity‚0.008333333767950535 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.2222222238779068 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 *jet/accuracy_1_of_2‚0.7377653121948242 7 Purity/*FRt*SRqqt4FBt/FBt_purity‚0.8333333134651184 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.4166666567325592 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.23530003428459167 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 *"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.5 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.4899425506591797 :"Purity/*FRt0SRqqt2FBt/event_purity‚0.26025593280792236 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9706541895866394 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.3697487711906433 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 ;"Purity/1FRt1SRqqt1FBt/event_purity‚0.047168973833322525 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.4322513937950134 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 8"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.574305534362793 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.36307236552238464 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 +jet/accuracy_2_of_3‚0.24079462885856628 particle/accuracy_3_of_3‚1 + Purity/1FRt3SRqqt1FBt/FRt_purity‚0.3125 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.2291666716337204 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 ;"Purity/1FRt1SRqqt2FBt/event_purity‚0.004166666883975267 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 particle/accuracy_2_of_3‚1 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 8 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.25648146867752075 :"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.30000001192092896 , Purity/*FRt2SRqqt2FBt/FRt_purity‚0.28125 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 jet/accuracy_6_of_7‚0 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.20909996330738068 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 :"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.44999998807907104 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.2857142984867096 jet/accuracy_1_of_7‚1 9"Purity/*FRt2SRqqt0FBt/event_purity‚0.0256629791110754 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 8"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.511054515838623 +particle/sensitivity‚0.3357149660587311 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.4285714328289032 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 ("Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0 _runtime‚5658.849324032 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.4538044333457947 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.09219858795404434 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.4183308184146881 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.4545099139213562 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 8 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.37291669845581055 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.3150291442871094 ("Purity/*FRt3SRqqt*FBt/event_purity‚0 :"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.44533976912498474 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.4465302526950836 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9864341616630554 ,jet/accuracy_3_of_3‚0.027545208111405373 particle/accuracy_2_of_6‚1 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9659090638160706 7 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.2666666805744171 & Purity/1FRt4SRqqt*FBt/FRt_purity‚0 8"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.949999988079071 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.3065249025821686 ;"Purity/1FRt2SRqqt1FBt/event_purity‚0.008620689623057842 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2738095223903656 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 jet/accuracy_3_of_7‚0 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9878169894218445 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.21987681090831757 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.20000000298023224 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 +"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.25 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9868918061256409 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.4346936047077179 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.8874269723892212 particle/accuracy_6_of_7‚1 * Purity/4FRt*SRqqt0FBt/FRt_purity‚0.125 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.1965513676404953 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 :"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.46128135919570923 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 ;"Purity/1FRt2SRqqt*FBt/event_purity‚0.002801120514050126 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9976190328598022 9"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.3768768012523651 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9966723322868347 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.06284764409065247 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.01367811020463705 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.19348660111427307 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 +"Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0.75 _step‚395 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.5263158082962036 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9820101857185364 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.4465302526950836 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9219577312469482 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9843736290931702 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.24029658734798431 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 ;"Purity/3FRt0SRqqt*FBt/event_purity‚0.011111111380159855 ) Purity/2FRt3SRqqt1FBt/FRt_purity‚0.25 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9963061213493347 :"Purity/*FRt1SRqqt2FBt/event_purity‚0.04203788936138153 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2587589621543884 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 *"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.5 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9743308424949646 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.3653067946434021 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9965516924858093 7 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.2081461250782013 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6893441081047058 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.29467183351516724 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9773243069648743 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.4217425286769867 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.4527410864830017 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 ,jet/accuracy_4_of_5‚0.007392473053187132 6validation_average_jet_accuracy‚0.4254087209701538 particle/accuracy_1_of_6‚1 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 particle/accuracy_2_of_5‚1 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.014040060341358185 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.030083145946264267 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 *"Purity/4FRt2SRqqt0FBt/SRqqt_purity‚0.5 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.3124999701976776 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.3194444477558136 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.8571428656578064 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 jet/accuracy_7_of_7‚0 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5206758975982666 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.24655170738697052 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.4854591190814972 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.4829787611961365 :"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.39559388160705566 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.2602807581424713 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.07630948722362518 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 *"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.5 6 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.987833559513092 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.5489911437034607 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 :"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.47629886865615845 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.7727272510528564 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9184028506278992 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 ( Purity/1FRt*SRqqt4FBt/FRt_purity‚0.5 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 6 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.994355320930481 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.1995239555835724 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.4195736348628998 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.20120622217655182 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 & Purity/1FRt2SRqqt2FBt/FBt_purity‚1 jet/accuracy_2_of_7‚0 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9787499904632568 6 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.529411792755127 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.02351347729563713 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.639604806900024®WZ“4 :"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.43829837441444397 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.7575758099555969 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.4341980218887329 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 7 Purity/*FRt3SRqqt*FBt/FBt_purity‚0.9285714030265808 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 particle/accuracy_6_of_6‚1 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9933333396911621 & Purity/1FRt3SRqqt2FBt/FRt_purity‚1 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 particle/accuracy_5_of_5‚1 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.9444444179534912 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8182422518730164 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9937499761581421 8"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.557539701461792 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 9"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.4094771444797516 & Purity/1FRt0SRqqt4FBt/FRt_purity‚1 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9907218813896179 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9966723322868347 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5491482615470886 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 particle/accuracy_1_of_1‚1 particle/accuracy_1_of_3‚1 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 8 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.49881696701049805 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 particle/accuracy_2_of_2‚1 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.09441624581813812 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 9"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.3829965889453888 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.20811952650547028 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 <"Purity/1FRt2SRqqt0FBt/event_purity‚0.0025012025143951178 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.24142533540725708 7 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.3484848737716675 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.4561403691768646 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 ( Purity/3FRt4SRqqt*FBt/FRt_purity‚0.5 & Purity/*FRt3SRqqt2FBt/FRt_purity‚1 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 :"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.45561301708221436 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 particle/accuracy_4_of_5‚1 /"Purity/0FRt0SRqqt3FBt/event_purity‚0.203125 :"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.47239184379577637 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.4097222089767456 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.17694681882858276 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.17210423946380615 :"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.45312222838401794 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 7 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.2155555784702301 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.4109366536140442 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.5315476059913635 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367‹äs ì“‹‚(:ˆ Ì­êÔ‡æý gpu.0.gpu‚55.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚176.738' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚44.1845 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0DA+¡hŒ‚(:Ý Ì­êÔûÐÖÿ' proc.memory.availableMB‚ 1952.421875 network.sent‚ 1.1778802e+07 disk.sdb.in‚0.828125! disk.sda2.out‚653.26220703125 disk.md0.in‚ 5.12890625 disk.rbd2.in‚0 cpu‚25.226954617399482 disk.sda.out‚653.26220703125 disk.rbd0.out‚ 1.404296875 disk.sda1.in‚0 disk.dm-0.in‚ 1102.05859375 proc.cpu.threads‚17) disk./.usagePercent‚75.83342374591106 disk.sda2.in‚ 4.30078125) proc.memory.percent‚18.32554340362549 disk.md0p1.out‚ 629.40234375$ disk./.usageGB‚2712.170551300049 disk.rbd0.in‚0 disk.nvme1n1.in‚ 498.61328125 disk.sdb1.out‚0 disk.sda.in‚ 4.30078125 disk.sdb1.in‚0! disk.sdb2.out‚653.26220703125 disk.dm-0.out‚ 66065.0625# memory_percent‚88.0833625793457 disk.md0p1.in‚ 5.12890625 disk.nvme0n1.out‚ 30741.9375 disk.rbd2.out‚ 1.341796875 disk.sdb2.in‚0.828125 disk.sda1.out‚0# proc.memory.rssMB‚ 3002.45703125 network.recv‚ 1.841322e+06 disk.sdb.out‚653.26220703125 disk.nvme1n1.out‚ 35323.125 disk.md0.out‚ 629.40234375 disk.nvme0n1.in‚ 603.4453125q 552- _wandbruntime‚5678  _runtime‚5678±Ë¶ôÆŽÀ $ _timestamp‚1.788516052967628e+09 _runtime‚5678.438096829 _step‚396 & lr-AdamW/pg1‚0.0006418799496182693 & lr-AdamW/pg2‚0.0006418799496182693 trainer/global_step‚4849{•+üËÅ _runtime‚5678.438096829 _step‚396 & lr-AdamW/pg1‚0.0006418799496182693 & lr-AdamW/pg2‚0.0006418799496182693 trainer/global_step‚4849 $ _timestamp‚1.788516052967628e+09ŒŽ!ÆÓ2- _wandbruntime‚5678  _runtime‚5678Ä­ûûù'‘óO JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚405253620627456 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚62408535 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚29247443 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955591168 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26088 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5521321863680 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚399732316766720 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚452957930672128 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚390767096281328 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚390767079241968 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 121800003 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚62408533 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚29246843 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚405254538038272 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 121826091 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 _step‚397 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955591168 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚91655978 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5766360169472 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 121826091 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 121800003 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚399732298763776 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 398533632 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5501638688186 $ _timestamp‚1.788516052968523e+09 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚396267820518734 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚405253620627456 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚447191810719744 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5766143892480 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚396268734969514 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26090 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5521321863680 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 222191599 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 100365506 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚600 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚399732298763776 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚447191786779648 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚399732316766720 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5522221271552 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 222165509 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410816 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 100365508 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚452958170889216 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 216276992 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407872 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1187 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚405253620627456 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚62408535 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚29247443 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955591168 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26088 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5521321863680 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚399732316766720 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚452957930672128 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚390767096281328 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚390767079241968 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 121800003 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚62408533 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚29246843 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚405254538038272 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 121826091 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 _step‚397 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955591168 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚91655978 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5766360169472 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 121826091 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 121800003 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚399732298763776 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 398533632 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5501638688186 $ _timestamp‚1.788516052968523e+09 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚396267820518734 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚405253620627456 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚447191810719744 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5766143892480 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚396268734969514 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26090 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5521321863680 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 222191599 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 100365506 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚600 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚399732298763776 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚447191786779648 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚399732316766720 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5522221271552 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 222165509 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410816 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 100365508 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚452958170889216 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 216276992 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407872 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1187 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 222236114 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚62420909 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 100385404 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 233040896 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 _runtime‚5679.419705592 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 222209998 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚390845076375456 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177047040 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚390845059336096 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 121824594 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚638 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5767290435584 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 trainer/global_step‚4849 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 222209998 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26114 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚447281106000384 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚396347823725168 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5501849938084 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5522434667008 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚453048396435968 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚91673908 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 222236114 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26114 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5523334088192 XJDeviceStatsŠúiRýMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917424128 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚405334363726336 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007241 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5767523476480 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚91673268 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 121850708 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177047040 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 256980992 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 % _timestamp‚1.7885160539485364e+09 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450988 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411628 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5522434667008 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _step‚398 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚405335281150464 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128zqŠÝ '•ƒN MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26114 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚447281106000384 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚396347823725168 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5501849938084 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5522434667008 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚453048396435968 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚91673908 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 222236114 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26114 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5523334088192 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917424128 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚405334363726336 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007241 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5767523476480 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚91673268 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 121850708 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177047040 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 256980992 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 % _timestamp‚1.7885160539485364e+09 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450988 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411628 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5522434667008 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _step‚398 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚405335281150464 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚399811947062272 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1187 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5523334088192 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 100385406 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚396346909274180 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 222236114 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚62420909 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 100385404 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 233040896 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 _runtime‚5679.419705592 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 222209998 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚390845076375456 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177047040 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚390845059336096 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 121824594 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚638 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5767290435584 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 trainer/global_step‚4849 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 222209998Ž¶Ä”T2–- _wandbruntime‚5679  _runtime‚5679ž|U—Ï /loss/FRt4/assignment_loss‚2.769652096124796 /loss/FRt2/detection_loss‚1.6455997980453752 1loss/SRqqt3/detection_loss‚1.8076112614944577 trainer/global_step‚4849 _step‚399 0loss/FBt1/assignment_loss‚0.3204123840041575 1loss/FBt2/assignment_loss‚0.15674252354770976 .loss/FRt4/detection_loss‚2.925302484860787 .loss/FBt4/detection_loss‚2.544459515147739 .loss/FBt1/detection_loss‚2.370408870279789 /loss/FBt2/detection_loss‚0.5572835461922235 % _timestamp‚1.7885160539506786e+09 2loss/SRqqt1/assignment_loss‚1.7009008623903892 %loss/total_loss‚40.25038085445521 /loss/FRt1/detection_loss‚2.2929495096945094 _runtime‚5679.420876978 .loss/FRt1/assignment_loss‚2.81232407041218 1loss/SRqqt2/assignment_loss‚1.112302364048655 .loss/FRt3/detection_loss‚2.529002178937961 /loss/FBt3/detection_loss‚2.2844761461019516 1loss/SRqqt3/assignment_loss‚1.543887955757479 1loss/SRqqt4/assignment_loss‚1.634254685137421 1loss/SRqqt2/detection_loss‚0.8262415295008754 epoch‚4 0loss/FBt3/assignment_loss‚0.2975502997287549 1loss/FBt4/assignment_loss‚0.07606257327521841 1loss/SRqqt1/detection_loss‚1.7430118217331465 0loss/SRqqt4/detection_loss‚2.252916668355465 0loss/FRt2/assignment_loss‚1.4177323926578869 /loss/FRt3/assignment_loss‚2.629295317026285D’‡\Z˜Ô 2loss/SRqqt1/assignment_loss‚1.7009008623903892 %loss/total_loss‚40.25038085445521 /loss/FRt1/detection_loss‚2.2929495096945094 _runtime‚5679.420876978 .loss/FRt1/assignment_loss‚2.81232407041218 1loss/SRqqt2/assignment_loss‚1.112302364048655 .loss/FRt3/detection_loss‚2.529002178937961 /loss/FBt3/detection_loss‚2.2844761461019516 1loss/SRqqt3/assignment_loss‚1.543887955757479 1loss/SRqqt4/assignment_loss‚1.634254685137421 1loss/SRqqt2/detection_loss‚0.8262415295008754 epoch‚4 0loss/FBt3/assignment_loss‚0.2975502997287549 1loss/FBt4/assignment_loss‚0.07606257327521841 1loss/SRqqt1/detection_loss‚1.7430118217331465 0loss/SRqqt4/detection_loss‚2.252916668355465 0loss/FRt2/assignment_loss‚1.4177323926578869 /loss/FRt3/assignment_loss‚2.629295317026285 /loss/FRt4/assignment_loss‚2.769652096124796 /loss/FRt2/detection_loss‚1.6455997980453752 1loss/SRqqt3/detection_loss‚1.8076112614944577 trainer/global_step‚4849 _step‚399 0loss/FBt1/assignment_loss‚0.3204123840041575 1loss/FBt2/assignment_loss‚0.15674252354770976 .loss/FRt4/detection_loss‚2.925302484860787 .loss/FBt4/detection_loss‚2.544459515147739 .loss/FBt1/detection_loss‚2.370408870279789 /loss/FBt2/detection_loss‚0.5572835461922235 % _timestamp‚1.7885160539506786e+09¾Ü ‰’™‚(:‡ Û­êÔƒâäþ gpu.0.gpu‚32.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚164.372' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚41.093 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0=Ûßxrš‚(:ç Û­êÔ‹–…ÿ! disk.sdb2.out‚654.83837890625 disk.sdb.out‚654.83837890625) proc.memory.availableMB‚ 1952.41796875 disk.sdb2.in‚0.828125 disk.rbd2.out‚ 1.341796875 network.recv‚ 1.846347e+06) disk./.usagePercent‚75.83622326489767% disk./.usageGB‚2712.2706756591797" disk.nvme1n1.out‚ 35394.4453125 disk.sdb1.in‚0 disk.md0.out‚ 630.91796875! disk.sda2.out‚654.83837890625$ memory_percent‚88.08338642120361 disk.sda1.in‚0 disk.nvme0n1.in‚ 603.4453125 disk.sdb.in‚0.828125 disk.sda.in‚ 4.30078125 disk.rbd0.in‚0 disk.sdb1.out‚0 disk.sda1.out‚0 disk.dm-0.out‚ 66136.3828125 disk.nvme0n1.out‚ 30741.9375 cpu‚25.22645201157017 disk.rbd2.in‚0 disk.md0p1.in‚ 5.12890625 disk.md0p1.out‚ 630.91796875 disk.md0.in‚ 5.12890625 disk.sda2.in‚ 4.30078125 proc.cpu.threads‚17 network.sent‚ 1.1889179e+07 disk.rbd0.out‚ 1.404296875 disk.nvme1n1.in‚ 498.61328125 disk.dm-0.in‚ 1102.05859375 disk.sda.out‚654.83837890625) proc.memory.percent‚18.32554340362549# proc.memory.rssMB‚ 3002.45703125ÊX€”›‚(:‰ ê­êÔ¯´Ê‚ gpu.0.gpu‚56.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚36.0 gpu.0.powerWatts‚115.357' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.83925 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0}ämpœ‚(:å ê­êÔŽ«† disk.sdb1.in‚0 disk.md0.in‚ 5.12890625) disk./.usagePercent‚75.83795831177369 disk.sdb2.in‚0.828125 disk.sdb.in‚0.828125 disk.rbd0.in‚0 disk.rbd2.in‚0 disk.md0p1.out‚ 632.390625 disk.nvme1n1.in‚ 498.61328125 disk.rbd2.out‚ 1.341796875# proc.memory.rssMB‚ 3002.45703125 proc.cpu.threads‚17# memory_percent‚88.0868911743164 disk.md0.out‚ 632.390625 network.sent‚ 1.1999618e+07 disk.md0p1.in‚ 5.12890625 disk.nvme0n1.in‚ 603.4453125# disk.nvme0n1.out‚31451.30078125" disk.nvme1n1.out‚ 35465.4765625 disk.dm-0.in‚ 1102.05859375! disk.sda2.out‚656.36181640625& proc.memory.availableMB‚ 1951.84375 cpu‚25.225923465436118 disk.sdb.out‚656.36181640625 disk.sda2.in‚ 4.30078125 disk.rbd0.out‚ 1.404296875% disk./.usageGB‚2712.3327293395996! disk.sdb2.out‚656.36181640625 disk.sda.in‚ 4.30078125) proc.memory.percent‚18.32554340362549 disk.sdb1.out‚0 network.recv‚ 1.852078e+06 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-0.out‚66916.77734375 disk.sda.out‚656.36181640625FJÍd’‚(:‡ ù­êÔ³Äæ€ gpu.0.gpu‚48.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚171.168' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.792 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0*o Ílž‚(:á ù­êÔè‡Ô‚ disk.sda2.in‚ 4.30078125 cpu‚25.22545324931187 disk.sda1.out‚0 disk.dm-0.in‚ 1102.2421875 network.sent‚ 1.2002714e+07 disk.rbd2.in‚0 disk.md0p1.in‚ 5.12890625& proc.memory.availableMB‚ 1952.09375 disk.dm-0.out‚ 67089.5234375 network.recv‚ 1.855435e+06$ disk./.usageGB‚2712.582534790039 disk.rbd2.out‚ 1.341796875 disk.nvme1n1.out‚ 35569.78125 disk.sdb1.in‚0 disk.md0p1.out‚ 634.51953125" disk.nvme0n1.out‚ 31519.7421875 disk.sdb.in‚0.828125 disk.nvme1n1.in‚ 498.796875 disk.sdb2.in‚0.828125 disk.nvme0n1.in‚ 603.4453125 disk.sdb.out‚658.55517578125 disk.sda.out‚658.55517578125! disk.sda2.out‚658.55517578125 disk.rbd0.in‚0 disk.sdb1.out‚0 disk.md0.in‚ 5.12890625 disk.rbd0.out‚ 1.404296875 disk.md0.out‚ 634.51953125$ memory_percent‚88.08536529541016) proc.memory.percent‚18.32554340362549 proc.cpu.threads‚17) disk./.usagePercent‚75.84494297671968! disk.sdb2.out‚658.55517578125# proc.memory.rssMB‚ 3002.45703125 disk.sda1.in‚0 disk.sda.in‚ 4.30078125’hePoŸ‚(:ä ˆ®êÔ•¹€ disk.dm-0.in‚ 1102.2421875 disk.md0p1.in‚ 5.12890625 disk.sda1.in‚0 proc.cpu.threads‚17 network.recv‚ 1.860053e+06$ disk./.usageGB‚2712.832534790039 disk.sdb1.out‚0$ memory_percent‚88.08610439300537) proc.memory.availableMB‚ 1951.97265625# proc.memory.rssMB‚ 3002.45703125 network.sent‚ 1.2011019e+07) disk./.usagePercent‚75.85193308135281 disk.sdb1.in‚0 disk.md0.out‚ 635.921875 disk.rbd2.in‚0 disk.sdb.in‚0.828125) proc.memory.percent‚18.32554340362549 disk.sda1.out‚0 disk.md0p1.out‚ 635.921875 cpu‚25.22484218176998 disk.md0.in‚ 5.12890625 disk.rbd2.out‚ 1.341796875 disk.nvme0n1.in‚ 603.4453125# disk.nvme0n1.out‚31608.72265625 disk.sda.in‚ 4.30078125 disk.sda.out‚660.03173828125# disk.nvme1n1.out‚35720.25390625 disk.sdb2.in‚0.828125! disk.sdb2.out‚660.03173828125 disk.sdb.out‚660.03173828125 disk.sda2.in‚ 4.30078125 disk.rbd0.in‚0 disk.rbd0.out‚ 1.404296875 disk.nvme1n1.in‚ 498.796875 disk.dm-0.out‚ 67328.9765625! disk.sda2.out‚660.03173828125­íþþ ‚(:… ˆ®êÔô®Æ€ gpu.0.gpu‚7.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚37.0 gpu.0.powerWatts‚146.38' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚36.595 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ù°†2¡- _wandbruntime‚5735  _runtime‚5735r¨˜QÅ¢¿ % lr-AdamW/pg1‚0.000636484844316568 % lr-AdamW/pg2‚0.000636484844316568 trainer/global_step‚4899 % _timestamp‚1.7885161099088316e+09 _runtime‚5735.379170502 _step‚400bìÙ检 % lr-AdamW/pg1‚0.000636484844316568 % lr-AdamW/pg2‚0.000636484844316568 trainer/global_step‚4899 % _timestamp‚1.7885161099088316e+09 _runtime‚5735.379170502 _step‚400Ÿ@Y2¤- _wandbruntime‚5735  _runtime‚5735oÂ웣!¥ôO QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚594 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚29528696 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚409291707396096 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚403713813541376 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚63027335 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚400222271882242 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 trainer/global_step‚4899 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26088 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1187 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 101360408 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 101360406 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚409291707396096 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955591168 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚92555435 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚394666083948368 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26090 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5577875851776 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚451657747816448 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚409290789985280 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955591168 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5577875851776 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚596 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚63027333 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚457481884258304 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚403713831544320 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 101360406 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26088 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5556187933874 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 123058993 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚451657771756544 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1212 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214665 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚394666100987728 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 224393311 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚403713831544320 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 224419401 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 123058993 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚92556031 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 398533632)¾ÿ'¦ùO ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 222568448 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 123032905 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5576976443904 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5823889933312 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007241 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5577875851776 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚451657747816448 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚409290789985280 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955591168 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5577875851776 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚596 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚63027333 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚457481884258304 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚403713831544320 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 101360406 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26088 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5556187933874 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 123058993 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚451657771756544 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1212 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214665 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚394666100987728 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 224393311 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚403713831544320 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 224419401 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 123058993 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚92556031 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 398533632 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚594 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚29528696 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚409291707396096 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚403713813541376 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚63027335 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚400222271882242 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 trainer/global_step‚4899 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26088 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1187 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 101360408 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 101360406 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚409291707396096 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955591168 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚92555435 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚394666083948368 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26090 JDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899421184 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚92573321 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5578988668416 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26114 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 123057496 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚409371533084160 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 224463916 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚409372450508288 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚400301360637688 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26114 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 226749440 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚394744064042496 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917424128 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 101380304 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899421184 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚92573922 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚457572118580736 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955591168 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5578988668416 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 224463916 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚451747067037184 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128Xí o '©ƒN EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899421184 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚92573321 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5578988668416 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26114 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 123057496 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚409371533084160 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 224463916 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚409372450508288 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚400301360637688 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26114 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 226749440 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚394744064042496 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917424128 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 101380304 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899421184 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚92573922 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚457572118580736 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955591168 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5578988668416 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 224463916 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚451747067037184 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 101380306 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 123083610 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984’ßn[2ª- _wandbruntime‚5736  _runtime‚5736>XÌ«Õ /loss/FRt1/assignment_loss‚2.321762161434821 2loss/SRqqt1/assignment_loss‚1.4765744895720092 /loss/FRt3/detection_loss‚2.1929331046753915 1loss/SRqqt3/detection_loss‚1.6321858364456105 _step‚403 0loss/FRt2/assignment_loss‚1.0741490480445681 1loss/SRqqt4/assignment_loss‚2.048693416174501 1loss/FBt2/assignment_loss‚0.07182186553903748 epoch‚4 _runtime‚5736.376467617 /loss/FRt4/assignment_loss‚3.803698701243247 /loss/FRt1/detection_loss‚1.9689399852305232 trainer/global_step‚4899 /loss/FRt3/assignment_loss‚2.458424545567611 1loss/SRqqt1/detection_loss‚1.6468234946493243 .loss/FBt2/detection_loss‚0.775086900678935 2loss/SRqqt2/assignment_loss‚1.0970949120754894 1loss/FBt1/assignment_loss‚??ÿ·S0.15235315375788228 1loss/SRqqt2/detection_loss‚1.1077662945931985 1loss/SRqqt4/detection_loss‚2.2513470593839884 /loss/FBt4/detection_loss‚2.1413213113943734 &loss/total_loss‚39.924217629145026 2loss/SRqqt3/assignment_loss‚0.9312153363705806 1loss/FBt3/assignment_loss‚0.22574172507574986 /loss/FRt4/detection_loss‚2.9528365865830453 .loss/FBt1/detection_loss‚2.461082463914698 /loss/FBt3/detection_loss‚3.2253867089748383 0loss/FBt4/assignment_loss‚0.2429676696910595 -loss/FRt2/detection_loss‚1.66401085807454 % _timestamp‚1.7885161109051907e+09pÂå`¬Ú _step‚403 0loss/FRt2/assignment_loss‚1.0741490480445681 1loss/SRqqt4/assignment_loss‚2.048693416174501 1loss/FBt2/assignment_loss‚0.07182186553903748 epoch‚4 _runtime‚5736.376467617 /loss/FRt4/assignment_loss‚3.803698701243247 /loss/FRt1/detection_loss‚1.9689399852305232 trainer/global_step‚4899 /loss/FRt3/assignment_loss‚2.458424545567611 1loss/SRqqt1/detection_loss‚1.6468234946493243 .loss/FBt2/detection_loss‚0.775086900678935 2loss/SRqqt2/assignment_loss‚1.0970949120754894 1loss/FBt1/assignment_loss‚0.15235315375788228 1loss/SRqqt2/detection_loss‚1.1077662945931985 1loss/SRqqt4/detection_loss‚2.2513470593839884 /loss/FBt4/detection_loss‚2.1413213113943734 &loss/total_loss‚39.924217629145026 2loss/SRqqt3/assignment_loss‚0.9312153363705806 1loss/FBt3/assignment_loss‚0.22574172507574986 /loss/FRt4/detection_loss‚2.9528365865830453 .loss/FBt1/detection_loss‚2.461082463914698 /loss/FBt3/detection_loss‚3.2253867089748383 0loss/FBt4/assignment_loss‚0.2429676696910595 -loss/FRt2/detection_loss‚1.66401085807454 % _timestamp‚1.7885161109051907e+09 /loss/FRt1/assignment_loss‚2.321762161434821 2loss/SRqqt1/assignment_loss‚1.4765744895720092 /loss/FRt3/detection_loss‚2.1929331046753915 1loss/SRqqt3/detection_loss‚1.6321858364456105“"ú”­‚(:‰ —®êÔ¶áýý gpu.0.gpu‚27.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚172.395' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.09875 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Tš Ã\®‚(:Ñ —®êÔÔ°™ÿ disk.rbd0.in‚0 disk.rbd0.out‚ 1.404296875 proc.cpu.threads‚17 disk.sdb.out‚ 661.484375% proc.memory.availableMB‚ 1301.4375# disk.nvme1n1.out‚35720.35546875 disk.sdb1.in‚0 disk.sdb2.in‚0.828125) proc.memory.percent‚18.32554340362549 cpu‚25.224269171304897) disk./.usagePercent‚75.85892958561786$ disk./.usageGB‚2713.082763671875 disk.sdb.in‚0.828125 disk.nvme1n1.in‚ 498.796875 disk.rbd2.in‚0 disk.sda1.in‚0$ memory_percent‚92.05665588378906 network.sent‚ 1.2014662e+07 disk.nvme0n1.in‚ 603.4453125" disk.nvme0n1.out‚ 31608.7890625 disk.sdb1.out‚0 disk.dm-0.in‚ 1102.2421875 disk.md0p1.in‚ 5.12890625 disk.sda.out‚ 661.484375 disk.sda2.out‚ 661.484375 disk.md0.in‚ 5.12890625 disk.rbd2.out‚ 1.341796875 disk.sda1.out‚0 network.recv‚ 1.863956e+06 disk.md0p1.out‚ 637.31640625 disk.sda2.in‚ 4.30078125 disk.md0.out‚ 637.31640625 disk.sdb2.out‚ 661.484375 disk.dm-0.out‚67329.14453125# proc.memory.rssMB‚ 3002.45703125 disk.sda.in‚ 4.30078125`"óÉ”¯‚(:‰ ¦®êÔíÏËÿ gpu.0.gpu‚25.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚168.545' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.13625 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ïZNh°‚(:Ý ¦®êÔÈȯ€ disk.rbd2.in‚0 disk.sdb2.in‚0.828125 disk.sdb.in‚0.828125 cpu‚25.223676991138174 network.recv‚ 1.869805e+06$ disk./.usageGB‚2713.081687927246 disk.sdb1.out‚0 disk.sda1.out‚0 disk.dm-0.out‚ 67757.0390625 disk.nvme0n1.in‚ 603.4453125 disk.nvme0n1.out‚ 31786.375 disk.sda.in‚ 4.30078125 disk.sda.out‚663.02392578125# proc.memory.rssMB‚ 3002.45703125 disk.rbd2.out‚ 1.341796875 network.sent‚ 1.2122645e+07$ memory_percent‚92.05775260925293 disk.sdb1.in‚0 disk.md0p1.in‚ 5.12890625( proc.memory.availableMB‚ 1301.2578125 disk.md0.in‚ 5.12890625 disk.md0.out‚ 638.796875 disk.nvme1n1.in‚ 498.796875" disk.nvme1n1.out‚ 35970.6640625 disk.md0p1.out‚ 638.796875 disk.sdb.out‚663.02392578125 disk.sda2.in‚ 4.30078125! disk.sda2.out‚663.02392578125! disk.sdb2.out‚663.02392578125) proc.memory.percent‚18.32554340362549( disk./.usagePercent‚75.8588995073478 disk.rbd0.in‚0 disk.rbd0.out‚ 1.404296875 disk.sda1.in‚0 disk.dm-0.in‚ 1102.2421875 proc.cpu.threads‚17N±¥”±‚(:‰ µ®êÔöÐàÿ gpu.0.gpu‚48.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚36.0 gpu.0.powerWatts‚115.025' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.75625 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0•Ý£m²‚(:â µ®êÔ뺤€ proc.cpu.threads‚17 disk.md0p1.out‚ 640.58203125 disk.sda.in‚ 4.30078125 disk.rbd0.in‚0 disk.nvme1n1.in‚ 498.796875 disk.md0.out‚ 640.58203125# proc.memory.rssMB‚ 3002.45703125 disk.sdb.in‚0.828125 disk.sdb.out‚664.84814453125 disk.dm-0.in‚ 1102.2421875 disk.md0p1.in‚ 5.12890625 disk.sda2.in‚ 4.30078125 disk.rbd0.out‚ 1.404296875 disk.rbd2.in‚0$ memory_percent‚92.05780029296875 cpu‚25.223142470744875 network.sent‚ 1.2130123e+07 disk.dm-0.out‚68005.21484375 disk.nvme0n1.in‚ 603.4453125 disk.sda.out‚664.84814453125 disk.sdb1.out‚0% disk./.usageGB‚2712.8096466064453 disk.sda1.in‚0 disk.sda1.out‚0) proc.memory.percent‚18.32554340362549# disk.nvme1n1.out‚36170.74609375 disk.sdb1.in‚0 disk.md0.in‚ 5.12890625! disk.sdb2.out‚664.84814453125! disk.sda2.out‚664.84814453125 disk.sdb2.in‚0.828125 network.recv‚ 1.873147e+06) disk./.usagePercent‚75.85129311816007 disk.nvme0n1.out‚ 31834.46875 disk.rbd2.out‚ 1.341796875# proc.memory.availableMB‚1301.25‹Ô’O’³‚(:‡ Ä®êÔ¿¯ÿ gpu.0.gpu‚41.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚37.0 gpu.0.powerWatts‚166.548' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚41.637 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ¢ÏÊa´‚(:Ö Ä®êÔÇ£ˆ€ network.sent‚ 1.2133751e+07$ disk./.usageGB‚2712.809341430664 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.sdb.out‚666.59912109375 disk.sda2.in‚ 4.30078125 disk.sda.out‚666.59912109375# disk.nvme1n1.out‚36287.11328125 network.recv‚ 1.877153e+06 disk.md0p1.out‚642.25# disk.nvme0n1.out‚31872.12109375 proc.cpu.threads‚17 disk.rbd2.in‚0! disk.sdb2.out‚666.59912109375 disk.sda1.out‚0 disk.dm-0.out‚ 68159.234375( disk./.usagePercent‚75.8512845853175 disk.md0p1.in‚ 5.12890625! disk.sda2.out‚666.59912109375 disk.rbd0.out‚ 1.404296875 disk.nvme1n1.in‚ 498.796875 disk.sdb2.in‚0.828125 disk.sda1.in‚0 disk.dm-0.in‚ 1102.2421875$ proc.memory.availableMB‚1301.375# proc.memory.rssMB‚ 3002.45703125 cpu‚25.222568839261925 disk.rbd2.out‚ 1.341796875 disk.nvme0n1.in‚ 603.4453125 disk.sdb.in‚0.828125$ memory_percent‚92.05703735351562 disk.md0.in‚ 5.12890625 disk.md0.out‚642.25 disk.sda.in‚ 4.30078125 disk.rbd0.in‚0) proc.memory.percent‚18.32554340362549ke2µ- _wandbruntime‚5793  _runtime‚5793³=! ǶÁ & lr-AdamW/pg1‚0.0006310464682870834 & lr-AdamW/pg2‚0.0006310464682870834 trainer/global_step‚4949 % _timestamp‚1.7885161677265942e+09 _runtime‚5793.196983662 _step‚404/ôœèÌ·Æ & lr-AdamW/pg2‚0.0006310464682870834 trainer/global_step‚4949 % _timestamp‚1.7885161677265942e+09 _runtime‚5793.196983662 _step‚404 & lr-AdamW/pg1‚0.0006310464682870834”Ÿ)f2¸- _wandbruntime‚5793  _runtime‚5793»?UŽù'¹óO OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26088 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 124265807 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚413327959343104 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26090 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 124265807 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410816 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚407695328318976 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 252800000 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 $ _timestamp‚1.788516167727672e+09 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5633530432000 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚462005474402304 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚616 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚456123732793344 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5612532002402 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 102355308 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5633530432000 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26088 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450780 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 102355306 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5632631024128 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚5793.198392768 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚407695328318976 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407872 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚413327959343104 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 226647203 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407872 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚456123708853248 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚398565105694128 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚93457260 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214665 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5632631024128 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚29810511 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚462005727202304 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5881994408960 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚398565088654768 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917410816 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177047040 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚614 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 124265807 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410816 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚407695328318976 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 252800000 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 $ _timestamp‚1.788516167727672e+09 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5633530432000 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚462005474402304 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚616 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚456123732793344 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5612532002402 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 102355308 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5633530432000 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26088 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450780 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 102355306 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5632631024128 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚5793.198392768 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚407695328318976 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407872 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚413327959343104 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 226647203 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407872 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚456123708853248 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚398565105694128 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚93457260 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214665 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5632631024128 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚29810511 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚462005727202304 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5881994408960 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚398565088654768 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917410816 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177047040 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚614 JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5633743827456 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚5794.360577954 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚413409619866112 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 226665602 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚63658511 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚63658509 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 124290398 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 _step‚406 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899421184 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5634643248640 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899421184 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450988 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚398643085788256 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007241 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 398533632 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411628 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177047040 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚93474563 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955591168 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚407774976617472 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚29816663 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5633743827456 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚462095947474432 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚456213028073984 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚407774958614528 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚407774976617472 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 226691718 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 226665602 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26114 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚398643068748896 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5613640663928 trainer/global_step‚4949 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917424128 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 102375206 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚609 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 % _timestamp‚1.7885161688880224e+09 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 226691718 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 228846592 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚413409619866112 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31686 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 102375204 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚462096200261120 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0j¼k# '½ƒN =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚5794.360577954 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚413409619866112 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 226665602 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚63658511 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚63658509 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 124290398 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 _step‚406 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899421184 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5634643248640 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899421184 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450988 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚398643085788256 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007241 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 398533632 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411628 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177047040 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚93474563 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955591168 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚407774976617472 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚29816663 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5633743827456 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚462095947474432 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚456213028073984 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚407774958614528 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚407774976617472 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 226691718 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 226665602 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26114 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚398643068748896 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5613640663928 trainer/global_step‚4949 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917424128 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 102375206 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚609 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 % _timestamp‚1.7885161688880224e+09 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 226691718 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 228846592 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚413409619866112 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31686 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 102375204 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚462096200261120 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 102375206 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚93475174 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚611 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚29816054 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚404256726452184 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5633743827456 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0–ÊB;Œ2¾- _wandbruntime‚5794  _runtime‚5794x{`¿Ú .loss/FBt3/detection_loss‚4.037184129158656 epoch‚4 0loss/FRt3/assignment_loss‚2.6529204036508287 2loss/SRqqt4/assignment_loss‚1.6361039218402678 1loss/SRqqt2/detection_loss‚0.9676287676621642 0loss/SRqqt4/detection_loss‚2.342288238386954 /loss/FRt3/detection_loss‚2.3569888582541827 2loss/SRqqt2/assignment_loss‚0.8576356692356057 1loss/FBt1/assignment_loss‚0.03173779171263883 /loss/FBt2/detection_loss‚0.7624303491012303 %loss/total_loss‚42.09298393746565 % _timestamp‚1.7885161688917155e+09 1loss/FBt2/assignment_loss‚0.11156475073392906 2loss/FBt4/assignment_loss‚0.009431125219685378 0loss/SRqqt1/detection_loss‚1.916974674165249 _step‚407 .loss/FBt4/detection_loss‚2.826505568292406 trainer/global_step‚4949 /loss/FRt4/assignment_loss‚3.376600233408121 2loss/SRqqt1/assignment_loss‚1.3616501711308957 /loss/FRt1/detection_loss‚1.7753339722111814 1loss/SRqqt3/detection_loss‚1.8290815575325743 0loss/FRt2/assignment_loss‚1.5286904999188013 2loss/SRqqt3/assignment_loss‚1.3499965008467119 /loss/FRt4/detection_loss‚3.0527811096264768 /loss/FBt1/detection_loss‚2.9640551110108695 _runtime‚5794.362054371 /loss/FRt2/detection_loss‚1.6574123527056404 0loss/FRt1/assignment_loss‚2.6574443886677424 2loss/FBt3/assignment_loss‚0.030543792992830276UƒeÀß trainer/global_step‚4949 /loss/FRt4/assignment_loss‚3.376600233408121 2loss/SRqqt1/assignment_loss‚1.3616501711308957 /loss/FRt1/detection_loss‚1.7753339722111814 1loss/SRqqt3/detection_loss‚1.8290815575325743 0loss/FRt2/assignment_loss‚1.5286904999188013 2loss/SRqqt3/assignment_loss‚1.3499965008467119 /loss/FRt4/detection_loss‚3.0527811096264768 /loss/FBt1/detection_loss‚2.9640551110108695 _runtime‚5794.362054371 /loss/FRt2/detection_loss‚1.6574123527056404 0loss/FRt1/assignment_loss‚2.6574443886677424 2loss/FBt3/assignment_loss‚0.030543792992830276 .loss/FBt3/detection_loss‚4.037184129158656 epoch‚4 0loss/FRt3/assignment_loss‚2.6529204036508287 2loss/SRqqt4/assignment_loss‚1.6361039218402678 1loss/SRqqt2/detection_loss‚0.9676287676621642 0loss/SRqqt4/detection_loss‚2.342288238386954 /loss/FRt3/detection_loss‚2.3569888582541827 2loss/SRqqt2/assignment_loss‚0.8576356692356057 1loss/FBt1/assignment_loss‚0.03173779171263883 /loss/FBt2/detection_loss‚0.7624303491012303 %loss/total_loss‚42.09298393746565 % _timestamp‚1.7885161688917155e+09 1loss/FBt2/assignment_loss‚0.11156475073392906 2loss/FBt4/assignment_loss‚0.009431125219685378 0loss/SRqqt1/detection_loss‚1.916974674165249 _step‚407 .loss/FBt4/detection_loss‚2.826505568292406—È`?Ì”Á‚(:‰ Ó®êÔ£°øý gpu.0.gpu‚55.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚176.461' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚44.11525 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Q$¿óo‚(:ä Ó®êÔÈŽ™€ network.recv‚ 1.881013e+06 disk.sdb.out‚667.97998046875 disk.sda.out‚667.97998046875 disk.sdb1.in‚0 network.sent‚ 1.2138951e+07! disk.sda2.out‚667.97998046875$ memory_percent‚92.05615520477295) proc.memory.availableMB‚ 1301.51953125# proc.memory.rssMB‚ 3002.45703125" disk.nvme0n1.out‚ 31872.5859375 disk.sda.in‚ 4.30078125 disk.md0.in‚ 5.12890625 disk.rbd2.in‚0 disk.rbd2.out‚ 1.341796875 disk.sdb2.in‚0.828125 disk.sda1.out‚0) disk./.usagePercent‚75.85186641852023 disk.sda2.in‚ 4.30078125! disk.nvme1n1.out‚ 36431.640625 disk.md0p1.in‚ 5.12890625 disk.nvme0n1.in‚ 603.4453125 disk.dm-0.out‚ 68304.2265625 cpu‚25.222037929488703$ disk./.usageGB‚2712.830150604248 disk.sdb1.out‚0 disk.sdb.in‚0.828125 disk.nvme1n1.in‚ 498.796875! disk.sdb2.out‚667.97998046875 disk.sda1.in‚0 disk.md0p1.out‚ 643.5859375 proc.cpu.threads‚17 disk.rbd0.in‚0 disk.rbd0.out‚ 1.404296875 disk.md0.out‚ 643.5859375 disk.dm-0.in‚ 1102.2421875) proc.memory.percent‚18.32554340362549䜃’Â(:‡ â®êÔ»Û½ÿ gpu.0.gpu‚13.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚39.0 gpu.0.powerWatts‚175.512' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.878 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ö¯iµrÄ‚(:ç â®êÔ…–ôÿ) disk./.usagePercent‚75.83989313382638 disk.rbd2.in‚0 disk.sda1.out‚0 disk.nvme0n1.in‚ 603.4453125 network.sent‚ 1.2246939e+07 network.recv‚ 1.886648e+06 disk.nvme1n1.in‚ 498.796875# disk.nvme1n1.out‚36628.11328125 disk.sda1.in‚0 disk.md0p1.out‚ 645.86328125 cpu‚25.2215563010832 disk.sdb.in‚0.828125 disk.rbd0.in‚0 disk.sdb1.out‚0 disk.md0.in‚ 5.12890625! disk.sdb2.out‚670.31689453125 disk.md0p1.in‚ 5.12890625 disk.sda.in‚ 4.30078125 disk.md0.out‚ 645.86328125 disk.sdb2.in‚0.828125 disk.dm-0.out‚68500.82421875$ memory_percent‚92.05625057220459# proc.memory.rssMB‚ 3002.45703125 disk.sdb.out‚670.31689453125 disk.sda.out‚670.31689453125 disk.sda2.in‚ 4.30078125 disk.sdb1.in‚0" disk.nvme0n1.out‚ 31872.7109375) proc.memory.availableMB‚ 1301.50390625) proc.memory.percent‚18.32554340362549 disk.rbd0.out‚ 1.404296875 disk.dm-0.in‚ 1102.2421875 proc.cpu.threads‚17$ disk./.usageGB‚2712.401927947998! disk.sda2.out‚670.31689453125 disk.rbd2.out‚ 1.341796875î9gÅ‚(:Ü ñ®êÔàð—ÿ disk.rbd0.in‚0 disk.sdb1.out‚0 disk.md0.out‚ 647.2265625 disk.rbd2.in‚0 disk.sdb2.out‚ 671.740234375) proc.memory.percent‚18.32554340362549 network.sent‚ 1.2252299e+07 network.recv‚ 1.890327e+06 disk.dm-0.in‚ 1102.2421875 disk.dm-0.out‚ 68629.1953125) proc.memory.availableMB‚ 1301.49609375 disk.sda.out‚ 671.740234375 disk.sda2.in‚ 4.30078125 disk.rbd0.out‚ 1.404296875 disk.md0.in‚ 5.12890625 disk.nvme0n1.in‚ 603.4453125$ memory_percent‚92.05629825592041 disk.sda1.out‚0 disk.sdb.out‚ 671.740234375 cpu‚25.220996110462792) disk./.usagePercent‚75.84337112045779% disk./.usageGB‚2712.5263175964355 disk.nvme1n1.in‚ 498.796875" disk.nvme1n1.out‚ 36756.4140625 disk.sda1.in‚0 disk.sda.in‚ 4.30078125 disk.sdb1.in‚0 disk.rbd2.out‚ 1.341796875 disk.sdb2.in‚0.828125 disk.sdb.in‚0.828125 disk.sda2.out‚ 671.740234375 disk.md0p1.in‚ 5.12890625 disk.md0p1.out‚ 647.2265625# proc.memory.rssMB‚ 3002.45703125 proc.cpu.threads‚17 disk.nvme0n1.out‚ 31872.78125"áÞ‡”Æ‚(:‰ ñ®êÔÅî¾ÿ gpu.0.gpu‚56.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚179.579' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚44.89475 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0…½€2Ç- _wandbruntime‚5845  _runtime‚5845“¤ÌÅÈ¿ _step‚408 % lr-AdamW/pg1‚0.000625565954879229 % lr-AdamW/pg2‚0.000625565954879229 trainer/global_step‚4999 % _timestamp‚1.7885162203443134e+09 _runtime‚5845.814710416ډϲÊÉÄ % lr-AdamW/pg1‚0.000625565954879229 % lr-AdamW/pg2‚0.000625565954879229 trainer/global_step‚4999 % _timestamp‚1.7885162203443134e+09 _runtime‚5845.814710416 _step‚408˜IÌy2Ê- _wandbruntime‚5845  _runtime‚5845‚×AÎf ËóO YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚411676843096576 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚411676861099520 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450780 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 103350206 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5688285604352 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411420 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚64264933 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 259091456 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214665 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5689185012224 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚30093715 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚460589669890048 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚417366046111744 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚466529446073344 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177047040 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚94358051 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 trainer/global_step‚4999 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 228848915 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 125524797 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚402464093361168 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚466529186981888 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26088 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5667081248090 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚408132089060038 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _runtime‚5845.816005622 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 103350208 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚94358650 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 $ _timestamp‚1.788516220345461e+09 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 103350208 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 125498709 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚597 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955591168 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚411676861099520 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1212 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5939517091840 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5667978659510 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1187 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5939752243200 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410816 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚409 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26088 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.Wh!Í“allocated‚ 5689185012224 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407872 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 103350206 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955591168 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5688285604352 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 235151360 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚417366046111744 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917410816 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚417365128700928 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚411676843096576 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚402464110400528 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407872 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 228875005 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 125524797 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0™)-þ'ÌøO ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5667081248090 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚408132089060038 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _runtime‚5845.816005622 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 103350208 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚94358650 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 $ _timestamp‚1.788516220345461e+09 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 103350208 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 125498709 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚597 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955591168 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚411676861099520 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1212 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5939517091840 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5667978659510 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1187 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5939752243200 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410816 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚409 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26088 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5689185012224 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407872 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 103350206 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955591168 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5688285604352 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 235151360 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚417366046111744 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917410816 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚417365128700928 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚411676843096576 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚402464110400528 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407872 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 228875005 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 125524797 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚411676843096576 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚411676861099520 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450780 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 103350206 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5688285604352 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411420 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚64264933 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 259091456 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214665 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5689185012224 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚30093715 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚460589669890048 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚417366046111744 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚466529446073344 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177047040 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚94358051 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 trainer/global_step‚4999 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 228848915 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 125524797 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚402464093361168 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚466529186981888 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26088™Œ”U2Í- _wandbruntime‚5846  _runtime‚5846áñ01'ÎþM WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚417446789223936 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 125523300 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 228919520 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚94376686 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚622 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚466619918936576 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5690297828864 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411628 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 103370106 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177047040 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26114 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 trainer/global_step‚4999 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 228893404 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 125549414 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚94376064 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚411756491395072 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899421184 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177047040 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 _step‚410 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚620 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 103370104 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007241 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚64277309 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 125549414 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955591168 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚30099375 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 228919520 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 103370104 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚417445871799808 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899421184 /q 'σN MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26114 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 trainer/global_step‚4999 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 228893404 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 125549414 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚94376064 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚411756491395072 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899421184 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177047040 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 _step‚410 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚620 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 103370104 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007241 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚64277309 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 125549414 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955591168 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚30099375 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 228919520 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 103370104 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚417445871799808 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899421184 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚417446789223936 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 125523300 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 228919520 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚94376686 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚622 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚466619918936576 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5690297828864 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411628 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 103370106 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177047040šs¼õ³2Ð- _wandbruntime‚5846  _runtime‚5846ÀôbÍÑÓ 0loss/FRt4/assignment_loss‚3.2356010861694813 0loss/SRqqt2/detection_loss‚1.209302563902358 /loss/FBt3/detection_loss‚3.1266108486387463 trainer/global_step‚4999 1loss/SRqqt1/assignment_loss‚1.222000800704552 2loss/SRqqt3/assignment_loss‚1.5654355536510305 .loss/FRt4/detection_loss‚2.925383945927024 epoch‚4 /loss/FRt2/assignment_loss‚1.372562669894912 1loss/FBt3/assignment_loss‚0.24307016785355195 0loss/SRqÞö"/Œqt1/detection_loss‚1.678490016798852 _runtime‚5846.785294808 1loss/FBt1/assignment_loss‚0.10676684224608834 .loss/FBt4/detection_loss‚2.302444252100858 /loss/FRt3/detection_loss‚2.3810384053398264 /loss/FBt2/detection_loss‚0.5513102314793147 0loss/FRt3/assignment_loss‚3.1128799698569556 2loss/SRqqt2/assignment_loss‚0.8832439349334634 1loss/FBt2/assignment_loss‚0.09084125382156634 /loss/FRt1/detection_loss‚1.9700424106151342 1loss/SRqqt3/detection_loss‚1.9099208116531372 /loss/FBt1/detection_loss‚2.6994393169879913 %loss/total_loss‚40.75311225336128 % _timestamp‚1.7885162213151681e+09 2loss/SRqqt4/assignment_loss‚1.4453882416471457 0loss/FBt4/assignment_loss‚0.1850093881895935 /loss/FRt2/detection_loss‚2.0193807262588632 0loss/SRqqt4/detection_loss‚2.335126898227594 _step‚411 .loss/FRt1/assignment_loss‚2.18182191646324•- ñ^ÒØ .loss/FRt4/detection_loss‚2.925383945927024 epoch‚4 /loss/FRt2/assignment_loss‚1.372562669894912 1loss/FBt3/assignment_loss‚0.24307016785355195 0loss/SRqqt1/detection_loss‚1.678490016798852 _runtime‚5846.785294808 1loss/FBt1/assignment_loss‚0.10676684224608834 .loss/FBt4/detection_loss‚2.302444252100858 /loss/FRt3/detection_loss‚2.3810384053398264 /loss/FBt2/detection_loss‚0.5513102314793147 0loss/FRt3/assignment_loss‚3.1128799698569556 2loss/SRqqt2/assignment_loss‚0.8832439349334634 1loss/FBt2/assignment_loss‚0.09084125382156634 /loss/FRt1/detection_loss‚1.9700424106151342 1loss/SRqqt3/detection_loss‚1.9099208116531372 /loss/FBt1/detection_loss‚2.6994393169879913 %loss/total_loss‚40.75311225336128 % _timestamp‚1.7885162213151681e+09 2loss/SRqqt4/assignment_loss‚1.4453882416471457 0loss/FBt4/assignment_loss‚0.1850093881895935 /loss/FRt2/detection_loss‚2.0193807262588632 0loss/SRqqt4/detection_loss‚2.335126898227594 _step‚411 .loss/FRt1/assignment_loss‚2.18182191646324 0loss/FRt4/assignment_loss‚3.2356010861694813 0loss/SRqqt2/detection_loss‚1.209302563902358 /loss/FBt3/detection_loss‚3.1266108486387463 trainer/global_step‚4999 1loss/SRqqt1/assignment_loss‚1.222000800704552 2loss/SRqqt3/assignment_loss‚1.5654355536510305›pâí4’Ó‚(:‡ €¯êÔÈ„äÿ gpu.0.gpu‚2.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚159.866' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚39.9665 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0òc mÔ‚(:â €¯êԴเ$ disk./.usageGB‚2712.526210784912 disk.sdb.out‚673.3076171875 disk.sda2.out‚673.3076171875 disk.nvme1n1.in‚ 498.796875 disk.md0.in‚ 5.12890625 disk.sda1.in‚0 disk.dm-0.out‚68763.84765625 proc.cpu.threads‚17 disk.md0p1.in‚ 5.12890625 disk.nvme0n1.in‚ 603.4453125 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.md0.out‚ 648.73046875$ memory_percent‚92.05844402313232 network.sent‚ 1.2271289e+07 disk.md0p1.out‚ 648.73046875# disk.nvme0n1.out‚31873.31640625 disk.sdb.in‚0.828125 disk.sda.in‚ 4.30078125 disk.nvme1n1.out‚ 36890.53125 disk.sdb2.in‚0.828125 network.recv‚ 1.894877e+06) proc.memory.percent‚18.32554340362549 disk.rbd2.in‚0 disk.sdb2.out‚673.3076171875 disk.sda1.out‚0 disk.dm-0.in‚ 1102.2421875 cpu‚25.220420205273715 disk.sda.out‚673.3076171875) proc.memory.availableMB‚ 1301.14453125( disk./.usagePercent‚75.8433681339629 disk.sda2.in‚ 4.30078125 disk.rbd0.in‚0 disk.rbd0.out‚ 1.416015625 disk.rbd2.out‚ 1.357421875# proc.memory.rssMB‚ 3002.45703125èõÓoÕ‚(:ä ¯êԥǮÿ) proc.memory.availableMB‚ 1301.62109375) proc.memory.percent‚18.32554340362549 proc.cpu.threads‚17) disk./.usagePercent‚75.84094971305757 disk.sda.out‚675.10498046875! disk.sda2.out‚675.10498046875 disk.sdb1.in‚0 network.sent‚ 1.2368149e+07 disk.sda.in‚ 4.30078125 disk.rbd2.in‚0 disk.sdb.out‚675.10498046875 disk.rbd0.in‚0 disk.md0.out‚ 650.46875 disk.sdb2.in‚0.828125 disk.dm-0.out‚68898.26171875 disk.sda2.in‚ 4.30078125 disk.rbd0.out‚ 1.416015625# disk.nvme1n1.out‚37024.87890625 disk.sda1.in‚0 disk.md0p1.out‚ 650.46875% disk./.usageGB‚2712.4397163391113 disk.sdb.in‚0.828125 disk.sdb1.out‚0$ memory_percent‚92.05553531646729 disk.md0p1.in‚ 5.12890625# proc.memory.rssMB‚ 3002.45703125 cpu‚25.219868632789215 disk.nvme1n1.in‚ 498.796875 network.recv‚ 1.899837e+06 disk.md0.in‚ 5.12890625 disk.rbd2.out‚ 1.419921875! disk.sdb2.out‚675.10498046875 disk.sda1.out‚0 disk.dm-0.in‚ 1102.2421875 disk.nvme0n1.in‚ 603.4453125" disk.nvme0n1.out‚ 31873.3828125P3~ܔւ(:‰ ¯êÔÍÁ«ÿ gpu.0.gpu‚51.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚170.559' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.63975 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0’ò¤Š“ׂ(:ˆ ž¯êÔ¸‹þ gpu.0.gpu‚53.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚165.598' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚41.3995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ŸÅ1ÁrØ‚(:ç ž¯êÔ‰ºÿ disk.sda1.in‚0 disk.dm-0.in‚ 1102.2421875 disk.sdb.out‚676.59033203125 disk.md0p1.in‚ 5.12890625( proc.memory.availableMB‚ 1300.8671875 cpu‚25.21939696199567 disk.sda.out‚676.59033203125 disk.rbd0.out‚ 1.478515625! disk.nvme1n1.out‚ 37214.640625 disk.sdb1.out‚0 disk.sdb2.in‚0.828125 disk.md0p1.out‚ 651.89453125 proc.cpu.threads‚17 disk.rbd0.in‚0 disk.nvme1n1.in‚ 498.796875 disk.sdb1.in‚0! disk.sdb2.out‚676.59033203125 disk.dm-0.out‚69088.19921875 disk.rbd2.out‚ 1.419921875 disk.sda1.out‚0) proc.memory.percent‚18.32554340362549 network.sent‚ 1.2372224e+07 disk.md0.in‚ 5.12890625 disk.md0.out‚ 651.89453125 network.recv‚ 1.904472e+06 disk.sda.in‚ 4.30078125$ memory_percent‚92.06013679504395 disk.sdb.in‚0.828125 disk.sda2.in‚ 4.30078125# proc.memory.rssMB‚ 3002.45703125) disk./.usagePercent‚75.84481306419156% disk./.usageGB‚2712.5778884887695 disk.nvme0n1.in‚ 603.4453125# disk.nvme0n1.out‚31873.55859375! disk.sda2.out‚676.59033203125 disk.rbd2.in‚0\$[”Ù‚(:‰ ­¯êÔ¸§‘þ gpu.0.gpu‚27.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚173.953' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.48825 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0@¢/übÚ‚(:× ­¯êÔÁîÖÿ disk.sda1.in‚0) proc.memory.availableMB‚ 1301.37109375 disk.sdb.in‚0.828125 disk.sdb1.out‚0 disk.rbd2.out‚ 1.50732421875# proc.memory.rssMB‚ 3002.45703125 network.sent‚ 1.2377525e+07$ disk./.usageGB‚2712.309856414795 disk.md0p1.out‚653.375 disk.nvme1n1.in‚ 498.796875 disk.md0.out‚653.375 disk.dm-0.out‚69151.37109375) proc.memory.percent‚18.32554340362549 cpu‚25.21892066105628 disk.md0p1.in‚ 5.12890625 disk.nvme0n1.in‚ 603.4453125 disk.sda2.out‚ 678.123046875 disk.rbd0.out‚ 1.478515625 disk.sda1.out‚0) disk./.usagePercent‚75.83731877522308 disk.sda2.in‚ 4.30078125 disk.sdb2.in‚0.828125 disk.sdb2.out‚ 678.123046875$ memory_percent‚92.05706119537354! disk.nvme0n1.out‚ 31873.640625 disk.sda.in‚ 4.30078125 disk.sda.out‚ 678.123046875 disk.sdb1.in‚0 disk.md0.in‚ 5.12890625 disk.sdb.out‚ 678.123046875# disk.nvme1n1.out‚37277.73046875 disk.rbd2.in‚0 disk.dm-0.in‚ 1102.2421875 proc.cpu.threads‚17 network.recv‚ 1.907694e+06 disk.rbd0.in‚0±Q´U2Û- _wandbruntime‚5900  _runtime‚5900’û½ÇÇÜÁ % _timestamp‚1.7885162746088333e+09 _runtime‚5900.079251523 _step‚412 & lr-AdamW/pg1‚0.0006200444462237842 & lr-AdamW/pg2‚0.0006200444462237842 trainer/global_step‚5049ÙÇu=ÌÝÆ & lr-AdamW/pg1‚0.0006200444462237842 & lr-AdamW/pg2‚0.0006200444462237842 trainer/global_step‚5049 % _timestamp‚1.7885162746088333e+09 _runtime‚5900.079251523 _step‚412œ-ÈŠ2Þ- _wandbruntime‚5900  _runtime‚5900qøÒ”ù'ßóO UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚421402298058752 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚465055654866944 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410816 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411420 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1187 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 398533632 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 104345108 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚95261575 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚95260971 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚30377238 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407872 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚421402298058752 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 126757699 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚406363098067568 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚415658357874176 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1212 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚64883733 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5744839592448 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 242314240 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚471053259715584 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚602 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26088ˆØ ¸þ'àøO ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 104345108 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚95261575 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚95260971 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚30377238 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407872 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚421402298058752 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 126757699 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚406363098067568 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚415658357874176 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1212 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚64883733 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5744839592448 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 242314240 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚471053259715584 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚602 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26088 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚421402298058752 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚465055654866944 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410816 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411420 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1187 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 398533632 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 231147322 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5999024090112 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 126782316 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚602 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 104365006 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚415738006172672 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚406441095201056 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚406441078161696 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚415738006172672 OADeviceStatsMonitor.on_train_batch_end/allocation.smalAô‰l_pool.freed‚ 126756202 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚471143755876864 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 398533632 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5745052987904 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917424128 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 231121206 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1187 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚95279094 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 126756202 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 242300928 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177047040 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 _runtime‚5901.143037769 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 104365004 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26114 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 218360832 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚415737988169728 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 104365006 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚30382985 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5745952409088 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 _step‚414 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5998805729280 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 231121206 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 231147322 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007241 % _timestamp‚1.7885162756719675e+09 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚421483041157632 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31686 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚465144974087680 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177047040 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 126782316 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚64896111 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚12125ö1 'ãƒN OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚600 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚412165629179200 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5724533978144 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚415737988169728 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899421184 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚95279696 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚30383585 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450988 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5745952409088 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚465144950147584 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411628 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚412164714728212 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚421483958581760 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚95279094 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 126756202 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 242300928 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177047040 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 _runtime‚5901.143037769 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 104365004 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26114 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 218360832 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚415737988169728 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 104365006 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚30382985 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5745952409088 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 _step‚414 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5998805729280 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 231121206 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 231147322 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007241 % _timestamp‚1.7885162756719675e+09 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚421483041157632 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31686 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚465144974087680 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177047040 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 126782316 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚64896111 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955591168 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899421184 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 trainer/global_step‚5049 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚421483958581760 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 104365004 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5723636566516 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 231147322 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5999024090112 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 126782316 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚602 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 104365006 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚415738006172672 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚406441095201056 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚406441078161696 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚415738006172672 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 126756202 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚471143755876864 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 398533632 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5745052987904 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917424128 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 231121206 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1187 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 Bˆœ ñóO M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚96164279 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚419639890654720 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955591168 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5800494172672 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6055717765632 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 105340006 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 127990601 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚475577552043520 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚30661746 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 105340008 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 127990601 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 105340008 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _runtime‚5954.496422447 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚639 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 218374144 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚425439467416576 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 _step‚417 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚637 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 127964513 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚30662383 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411420 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5799594764800 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177047040 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚469521615903744 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450780 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 242314240 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚425440384827392 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚65502533 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚475577309729280 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚419639890654720 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5778871973726 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5800494172672 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚96164918 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410816 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚425439467416576 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917410816 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955591168 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚469521591963648 $ _timestamp‚1.788516329025827e+09 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 105340006 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚425440384827392 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007241 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 127964513 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 233330609 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1187 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 trainer/global_step‚5099 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚419639872651776 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26088 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚65502535 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5777974562306 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.la­¯ÿ³Ùrge_pool.allocated‚ 13037993984 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚419639872651776 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚416040991787054 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 233304519 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚425439467416576 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917410816 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955591168 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚469521591963648 $ _timestamp‚1.788516329025827e+09 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 105340006 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚425440384827392 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007241 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 127964513 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 233330609 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1187 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 trainer/global_step‚5099 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚419639872651776 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26088 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚65502535 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5777974562306 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚419639872651776 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚416040991787054 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 233304519 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 105340006 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 127990601 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚475577552043520 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚30661746 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 105340008 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 127990601 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 105340008 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _runtime‚5954.496422447 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚639 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 218374144 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚425439467416576 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 _step‚417 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚637 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 127964513 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚30662383 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411420 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5799594764800 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177047040 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚469521615903744 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450780 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 242314240 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚425440384827392 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚65502533 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚475577309729280 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚419639890654720 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5778871973726 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5800494172672 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚96164918 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0¡'°~¨’ó‚(:‡ é¯êÔŸ“þ gpu.0.gpu‚41.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚151.95' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚37.9875 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0”»¿@pô‚(:å é¯êÔßõ–) disk./.usagePercent‚75.85890964009835 disk.md0.out‚ 659.640625 disk.sda1.out‚0 disk.md0p1.out‚ 659.640625 disk.nvme0n1.in‚ 603.4453125 proc.cpu.threads‚17 disk.nvme1n1.in‚ 498.796875 disk.rbd2.in‚0 disk.dm-0.out‚ 69862.3125( proc.memory.availableMB‚ 1301.1484375 network.recv‚ 1.924552e+06# disk.nvme0n1.out‚31873.98828125% disk./.usageGB‚2713.0820503234863 disk.rbd0.in‚0 disk.sda1.in‚0 disk.sda2.in‚ 4.30078125# disk.nvme1n1.out‚37988.32421875 disk.sdb1.out‚0 disk.rbd2.out‚ 1.63330078125 disk.sdb.out‚684.62841796875) proc.memory.percent‚18.32554340362549 disk.sdb.in‚0.828125 disk.sda.in‚ 4.30078125 disk.sda.out‚684.62841796875$ memory_percent‚92.05842018127441# proc.memory.rssMB‚ 3002.45703125 disk.rbd0.out‚ 1.62060546875 disk.md0p1.in‚ 5.12890625! disk.sda2.out‚684.62841796875 cpu‚25.21700364806395 disk.sdb1.in‚0 disk.md0.in‚ 5.12890625 disk.sdb2.in‚0.828125! disk.sdb2.out‚684.62841796875 disk.dm-0.in‚ 1102.2421875 network.sent‚ 1.2502776e+07 ;Û 2õ- _wandbruntime‚5955  _runtime‚5955ŒÕˆ¼'öþM WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚419719502947328 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚469610935124480 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 % _timestamp‚1.7885163300799038e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1187 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚410340082868096 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 233349008 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚419719502947328 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 128015218 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚65514911 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚30667510 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450988 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411628 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚425520210515456 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6056878505472 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚475668038282240 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚469610911184384 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚96182419 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5800707568128 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26114 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5801606989312 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31686 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 248592384 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 398533632 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 _step‚418 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 105359906 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚475667789689856 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚410340099907456 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚425521127939584 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚65514909 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007241 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 105359904 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚30668137 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 127989104 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚419719520950272 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899421184 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26114 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5779083223624 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177047040 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 233375124 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 233375124 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712áfdžÀ!÷ƒN SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899421184 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚96183048 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚416120080542708 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚475668038282240 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚469610911184384 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚96182419 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5800707568128 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26114 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5801606989312 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31686 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 248592384 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 398533632 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 _step‚418 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 105359906 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚475667789689856 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚410340099907456 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚425521127939584 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚65514909 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007241 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 105359904 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚30668137 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 127989104 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚419719520950272 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899421184 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26114 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5779083223624 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177047040 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 233375124 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 233375124 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚419719502947328 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚469610935124480 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 % _timestamp‚1.7885163300799038e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1187 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚410340082868096 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 233349008 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚419719502947328 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 128015218 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚65514911 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚30667510 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450988 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411628 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚425520210515456 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6056878505472 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚429476636774400 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚429477554185216 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 129197415 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚480101567900160 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5855249345024 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411420 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 trainer/global_step‚5149 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 106334908 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177047040 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚608 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1187 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214665 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 129223503 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917410816 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26088 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5833421219414 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407872 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚66121335 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚423621405432320 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5834318630834 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 106334908 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚97067737 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 398533632 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6113990959616 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007241 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26090 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955591168 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚610 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410816 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚429476636774400 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407872 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 106334906 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5856148752896 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚423621387429376 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 129197415 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6113780974080 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚419995443150562 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚473987553000448 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26088 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955591168 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚480101333974528 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚419994528699782 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚423621405432320 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚414161124519728 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚30947012 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _step‚421 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5856148752896 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 233925632 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚473987576940544 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚429477554185216 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚30946404 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450780 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 235558411 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚414161107480368 $ _timestamp‚1.788516381141405e+09 _runtime‚6006.611947885 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 129223503 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚97068347 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1212 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5855249345024 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚66121333 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚423621387429376 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 235532321 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 106334906 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 209985536ËÍ9æþ'†øO H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 233925632 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚473987576940544 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚429477554185216 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚30946404 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450780 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 235558411 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚414161107480368 $ _timestamp‚1.788516381141405e+09 _runtime‚6006.611947885 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 129223503 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚97068347 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1212 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5855249345024 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚66121333 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚423621387429376 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 235532321 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 106334906 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 209985536 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚429476636774400 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚429477554185216 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 129197415 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚480101567900160 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5855249345024 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411420 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 trainer/global_step‚5149 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 106334908 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177047040 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚608 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1187 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214665 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 129223503 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917410816 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26088 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5833421219414 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407872 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚66121335 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚423621405432320 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5834318630834 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 106334908 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚97067737 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 398533632 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6113990959616 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007241 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26090 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955591168 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚610 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410816 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚429476636774400 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407872 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 106334906 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5856148752896 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚423621387429376 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 129197415 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6113780974080 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚419995443150562 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚473987553000448 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26088 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955591168 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚480101333974528 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚419994528699782 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚423621405432320 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚414161124519728 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚30947012 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _step‚421 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5856148752896¥¢ÄkÏ2‡- _wandbruntime‚6007  _runtime‚60071rI' ˆýM E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 106354804 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 trainer/global_step‚5149 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚429558297297408 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚474076896161280 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31686 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚423701035727872 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚30951986 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚97086302 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177047040 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚605 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚480192058624000 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚480191807934464 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5857261569536 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 235576810 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚30952591 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6115162462720 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚YaLÜ 1182793728 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 129248120 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚474076872221184 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26114 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411628 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 $ _timestamp‚1.788516382199715e+09 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚420074531906216 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917424128 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚429557379873280 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚423701017724928 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5856362148352 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26114 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 235576810 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177047040 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 235602926 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚420073617455228 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚414239104613856 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚97085695 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚66133709 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 129248120 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5856362148352 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚423701035727872 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 398533632 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 235602926 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 106354804 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 129222006 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899421184 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0µLô'‰‚N L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 129248120 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚474076872221184 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26114 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411628 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 $ _timestamp‚1.788516382199715e+09 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚420074531906216 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917424128 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚429557379873280 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚423701017724928 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5856362148352 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26114 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 235576810 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177047040 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 235602926 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚420073617455228 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚414239104613856 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚97085695 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚66133709 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 129248120 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5856362148352 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚423701035727872 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 398533632 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 235602926 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 106354804 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 129222006 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899421184 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 106354804 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 trainer/global_step‚5149 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚429558297297408 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚474076896161280 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31686 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚423701035727872 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚30951986 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚97086302 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177047040 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚605 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚480192058624000 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚480191807934464 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5857261569536 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 235576810 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚30952591 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6115162462720 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728¦ 0'±2Š- _wandbruntime‚6007  _runtime‚6007ŸáF_‹Ù % _timestamp‚1.7885163822022038e+09 .loss/FRt4/detection_loss‚2.668172692259153 0loss/FRt3/assignment_loss‚2.5461241514594466 0loss/FRt4/assignment_loss‚3.8819905320803323 2loss/SRqqt4/assignment_loss‚1.9738926977703446 /loss/FRt2/detection_loss‚1.4369556170871907 1loss/FBt2/assignment_loss‚0.08660129000587505 1loss/SRqqt4/detection_loss‚2.3183760101857938 .loss/FBt4/detection_loss‚3.343926328420639 _step‚423 1loss/SRqqt2/detection_loss‚0.9338364666089033 1loss/SRqqt3/detection_loss‚1.6451170653677905 0loss/FBt2/detection_loss‚0.42160034164740345 .loss/FBt3/detection_loss‚2.295917756855488 %loss/total_loss‚41.99222874911715 /loss/FRt1/assignment_loss‚3.337338909214618 0loss/FRt2/assignment_loss‚1.0303415624718917 2loss/SRqqt2/assignment_loss‚1.0646002222783864 1loss/FBt3/assignment_loss‚0.18357383476541145 /loss/FRt3/detection_loss‚2.0422285251595356 _runtime‚ 6007.67235101 2loss/SRqqt1/assignment_loss‚1.4451778660342098 1loss/SRqqt1/detection_loss‚1.7355425226688386 trainer/global_step‚5149 /loss/FBt1/detection_loss‚3.8199427872896194 1loss/SRqqt3/assignment_loss‚1.688718107385704 2loss/FBt1/assignment_loss‚0.002990995468280744 1loss/FBt4/assignment_loss‚0.04976161278027576 /loss/FRt1/detection_loss‚2.0395008538520214 epoch‚4Ž¡Ä‰dŒÞ 1loss/SRqqt3/assignment_loss‚1.688718107385704 2loss/FBt1/assignment_loss‚0.002990995468280744 1loss/FBt4/assignment_loss‚0.04976161278027576 /loss/FRt1/detection_loss‚2.0395008538520214 epoch‚4 % _timestamp‚1.7885163822022038e+09 .loss/FRt4/detection_loss‚2.668172692259153 0loss/FRt3/assignment_loss‚2.5461241514594466 0loss/FRt4/assignment_loss‚3.8819905320803323 2loss/SRqqt4/assignment_loss‚1.9738926977703446 /loss/FRt2/detection_loss‚1.4369556170871907 1loss/FBt2/assignment_loss‚0.08660129000587505 1loss/SRqqt4/detection_loss‚2.3183760101857938 .loss/FBt4/detection_loss‚3.343926328420639 _step‚423 1loss/SRqqt2/detection_loss‚0.9338364666089033 1loss/SRqqt3/detection_loss‚1.6451170653677905 0loss/FBt2/detection_loss‚0.42160034164740345 .loss/FBt3/detection_loss‚2.295917756855488 %loss/total_loss‚41.99222874911715 /loss/FRt1/assignment_loss‚3.337338909214618 0loss/FRt2/assignment_loss‚1.0303415624718917 2loss/SRqqt2/assignment_loss‚1.0646002222783864 1loss/FBt3/assignment_loss‚0.18357383476541145 /loss/FRt3/detection_loss‚2.0422285251595356 _runtime‚ 6007.67235101 2loss/SRqqt1/assignment_loss‚1.4451778660342098 1loss/SRqqt1/detection_loss‚1.7355425226688386 trainer/global_step‚5149 /loss/FBt1/detection_loss‚3.8199427872896194§³-öÄ”‚(:‰ ¥°êÔõ‘üý gpu.0.gpu‚55.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚172.395' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.09875 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0„ðº÷rŽ‚(:ç ¥°êÔÒ†•€ disk.sda2.in‚ 4.30078125 disk.sda1.in‚0 disk.nvme0n1.in‚ 603.4453125 cpu‚25.215146663321303 disk.sdb2.in‚0.828125 disk.md0p1.out‚ 666.109375 disk.sda.in‚ 4.30078125! disk.sda2.out‚691.35693359375 disk.rbd0.in‚0 disk.sdb1.out‚0 disk.md0.in‚ 5.12890625 disk.dm-0.in‚ 1102.2421875 proc.cpu.threads‚17% disk./.usageGB‚2712.1333236694336 disk.rbd2.in‚0 disk.dm-0.out‚ 70739.6796875# disk.nvme0n1.out‚31874.86328125 disk.nvme1n1.in‚ 498.796875 disk.rbd0.out‚ 1.68359375# disk.nvme1n1.out‚38864.81640625 network.recv‚ 1.943035e+06) disk./.usagePercent‚75.83238284577808 disk.md0.out‚ 666.109375! disk.sdb2.out‚691.35693359375 disk.sdb.out‚691.35693359375 disk.sda.out‚691.35693359375 disk.sdb1.in‚0) proc.memory.availableMB‚ 1301.31640625# proc.memory.rssMB‚ 3002.45703125 network.sent‚ 1.2708215e+07 disk.md0p1.in‚ 5.12890625 disk.sdb.in‚0.828125) proc.memory.percent‚18.32554340362549 disk.rbd2.out‚ 1.63330078125 disk.sda1.out‚0$ memory_percent‚92.05739498138428%#V’‚(:‡ ´°êÔÒÛæÿ gpu.0.gpu‚43.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚167.604' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚41.901 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0/8Õ³h‚(:Ý ´°êÔ¾£›€ disk.sda.out‚693.26611328125 disk.rbd0.out‚ 1.68359375$ memory_percent‚92.05589294433594# proc.memory.rssMB‚ 3002.45703125 cpu‚25.214695842797024" disk.nvme0n1.out‚ 31875.1796875 disk.sda2.in‚ 4.30078125 disk.sdb1.out‚0% disk./.usageGB‚2712.2764282226562 disk.sda1.in‚0 disk.rbd0.in‚0 disk.sdb1.in‚0% proc.memory.availableMB‚ 1301.5625) proc.memory.percent‚18.32554340362549 disk.md0.in‚ 5.12890625 disk.md0.out‚ 667.9609375 disk.rbd2.in‚0 disk.dm-0.out‚70886.75 disk.sda.in‚ 4.30078125 disk.nvme1n1.in‚ 498.796875 disk.md0p1.in‚ 5.12890625 disk.md0p1.out‚ 667.9609375 disk.sdb.out‚693.26611328125! disk.sda2.out‚693.26611328125( disk./.usagePercent‚75.8363841089801 disk.rbd2.out‚ 1.63330078125! disk.sdb2.out‚693.26611328125 disk.dm-0.in‚ 1102.2421875 disk.nvme0n1.in‚ 603.4453125 network.recv‚ 1.948528e+06 disk.sda1.out‚0 disk.sdb.in‚0.828125" disk.nvme1n1.out‚ 39011.5703125 proc.cpu.threads‚17 network.sent‚ 1.2818282e+07 disk.sdb2.in‚0.828125VN’‘‚(:‡ ðêÔ®•†þ gpu.0.gpu‚47.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚114.408' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.602 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0O%žq’‚(:æ ðêÔ¥»Îÿ! disk.sda2.out‚694.75732421875 disk.nvme1n1.in‚ 498.796875 disk.sdb1.out‚0$ memory_percent‚92.05882549285889 disk.md0.in‚ 5.12890625 disk.md0.out‚ 669.37890625! disk.sdb2.out‚694.75732421875 disk.sda1.in‚0 disk.dm-0.out‚70931.25390625) proc.memory.percent‚18.32554340362549$ disk./.usageGB‚2712.401584625244 disk.sda.out‚694.75732421875 disk.dm-0.in‚ 1102.2421875 network.recv‚ 1.952824e+06 disk.sda2.in‚ 4.30078125 disk.rbd0.in‚0 disk.rbd0.out‚ 1.68359375 disk.md0p1.in‚ 5.12890625 cpu‚25.21417057959248) disk./.usagePercent‚75.83988353437849 disk.nvme0n1.out‚ 31875.1875# disk.nvme1n1.out‚39056.06640625 disk.rbd2.out‚ 1.63330078125 disk.md0p1.out‚ 669.37890625 disk.nvme0n1.in‚ 603.4453125 disk.sdb.in‚0.828125 disk.sda.in‚ 4.30078125 disk.sdb1.in‚0 disk.rbd2.in‚0) proc.memory.availableMB‚ 1301.08203125 network.sent‚ 1.2822579e+07 disk.sdb.out‚694.75732421875 disk.sdb2.in‚0.828125 proc.cpu.threads‚17 disk.sda1.out‚0# proc.memory.rssMB‚ 3002.45703125€°S2“- _wandbruntime‚6058  _runtime‚6058¶³6Æ”À _runtime‚ 6058.58929619 _step‚424 & lr-AdamW/pg1‚0.0006032454967894067 & lr-AdamW/pg2‚0.0006032454967894067 trainer/global_step‚5199 % _timestamp‚1.7885164331188562e+09¥ÎÉË•Å & lr-AdamW/pg1‚0.0006032454967894067 & lr-AdamW/pg2‚0.0006032454967894067 trainer/global_step‚5199 % _timestamp‚1.7885164331188562e+09 _runtime‚ 6058.58929619 _step‚424¨)øÄ2–- _wandbruntime‚6058  _runtime‚6058Tòa¿—ôO WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955591168 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 237786213 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 107329806 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 216276992 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5910903925248 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450780 _step‚425 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 107329806 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚433514723543040 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚31232338 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚31231740 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1187 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚484625350955520 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚478453514037248 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚427602920209920 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚433513806132224 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 130430317 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚433514723543040 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚433513806132224 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 130456405 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411420 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚97972473 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 130430317 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26088 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917410816 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407872 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚66740135 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26090 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214665 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 237760123 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 107329808 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007241 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 107329808 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177047040 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 trainer/global_step‚5199 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410816 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚427602902206976 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5910903925248 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 240217088 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1212 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚478453537977344 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 % _timestamp‚1.7885164331200433e+09 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚418060112186768 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5888867876522 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚600 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407872 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5911803333120 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26088 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6172053195264 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 107329806 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 216276992 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5910903925248 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450780 _step‚425 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 107329806 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚433514723543040 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚31232338 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚31231740 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1187 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚484625350955520 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚478453514037248 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚427602920209920 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚433513806132224 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 130430317 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚433514723543040 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚433513806132224 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 130456405 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411420 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚97972473 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 130430317 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26088 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917410816 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407872 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚66740135 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26090 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214665 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 237760123 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 107329808 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007241 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 107329808 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177047040 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 trainer/global_step‚5199 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410816 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚427602902206976 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5910903925248 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 240217088 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1212 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚478453537977344 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 % _timestamp‚1.7885164331200433e+09 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚418060112186768 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5888867876522 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚600 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407872 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5911803333120 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26088 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6172053195264 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 JDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 237830728 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5912916149760 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31686 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚484715841870848 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 237804612 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 130481022 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6173231168000 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚607 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 _step‚426 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1187 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚605 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 398533632 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 130454908 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955591168 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5912916149760 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26114 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚66752511 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚478542833257984 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177047040 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚433595466655232 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5912016728576 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚427682532502528 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 237830728 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚97989798 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚418138109320256 % _timestamp‚1.7885164340797935e+09 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6173008612864 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5912016728576 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚424028068818736 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 237804612 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚31237894 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450988 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚66752509 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚484716088366080 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 _runtime‚6059.550910181 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5890873949468 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5889976537840 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 130454908 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899421184 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177047040 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 107349706 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899421184 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564#q=•m&›ƒN REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚66752509 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚484716088366080 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 _runtime‚6059.550910181 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5890873949468 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5889976537840 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 130454908 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899421184 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177047040 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 107349706 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899421184 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 107349704 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 237830728 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5912916149760 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31686 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚484715841870848 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 237804612 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 130481022 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6173231168000 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚607 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 _step‚426 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1187 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚605 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 398533632 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 130454908 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955591168 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5912916149760 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26114 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚66752511 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚478542833257984 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177047040 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚433595466655232 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5912016728576 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚427682532502528 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 237830728 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚97989798 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚418138109320256 % _timestamp‚1.7885164340797935e+09 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6173008612864 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5912016728576 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚424028068818736 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 237804612 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚31237894 OADeo°LìœviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450988 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607ªµ¾²2œ- _wandbruntime‚6059  _runtime‚60599Ù‘>]× trainer/global_step‚5199 _runtime‚6059.551965646 epoch‚4 1loss/FBt2/assignment_loss‚0.14612247072957416 /loss/FRt2/detection_loss‚1.4273502115928938 ,loss/FBt3/detection_loss‚2.5627601146698 /loss/FBt4/detection_loss‚3.1043068170547485 2loss/SRqqt2/assignment_loss‚1.0609777829765032 2loss/SRqqt4/assignment_loss‚1.5022811688482762 1loss/FBt3/assignment_loss‚0.07611213614278256 % _timestamp‚1.7885164340818515e+09 0loss/FRt1/assignment_loss‚2.5737275852097405 /loss/FRt4/detection_loss‚3.0583896040916443 1loss/SRqqt1/detection_loss‚1.6322208519611094 1loss/SRqqt3/detection_loss‚1.8634682808816434 /loss/FRt4/assignment_loss‚3.108227779125345 2loss/SRqqt3/assignment_loss‚1.4848503209650517 1loss/FBt1/assignment_loss‚0.03375956760282861 /loss/FRt1/detection_loss‚2.0577868804711095 _step‚427 0loss/FRt2/assignment_loss‚1.2681926252349975 /loss/FRt3/detection_loss‚2.3510153183713554 /loss/FBt2/detection_loss‚0.6259527197573334 &loss/total_loss‚39.613861862229626 .loss/FRt3/assignment_loss‚2.90636750459671 0loss/FBt4/assignment_loss‚0.1444661267367857 1loss/SRqqt2/detection_loss‚0.9574721351058947 1loss/SRqqt4/detection_loss‚2.3774526201188566 2loss/SRqqt1/assignment_loss‚1.3092147811108994 /loss/FBt1/detection_loss‚1.9813864588737489„ß_bžÜ /loss/FBt1/detection_loss‚1.9813864588737489 trainer/global_step‚5199 _runtime‚6059.551965646 epoch‚4 1loss/FBt2/assignment_loss‚0.14612247072957416 /loss/FRt2/detection_loss‚1.4273502115928938 ,loss/FBt3/detection_loss‚2.5627601146698 /loss/FBt4/detection_loss‚3.1043068170547485 2loss/SRqqt2/assignment_loss‚1.0609777829765032 2loss/SRqqt4/assignment_loss‚1.5022811688482762 1loss/FBt3/assignment_loss‚0.07611213614278256 % _timestamp‚1.7885164340818515e+09 0loss/FRt1/assignment_loss‚2.5737275852097405 /loss/FRt4/detection_loss‚3.0583896040916443 1loss/SRqqt1/detection_loss‚1.6322208519611094 1loss/SRqqt3/detection_loss‚1.8634682808816434 /loss/FRt4/assignment_loss‚3.108227779125345 2loss/SRqqt3/assignment_loss‚1.4848503209650517 1loss/FBt1/assignment_loss‚0.03375956760282861 /loss/FRt1/detection_loss‚2.0577868804711095 _step‚427 0loss/FRt2/assignment_loss‚1.2681926252349975 /loss/FRt3/detection_loss‚2.3510153183713554 /loss/FBt2/detection_loss‚0.6259527197573334 &loss/total_loss‚39.613861862229626 .loss/FRt3/assignment_loss‚2.90636750459671 0loss/FBt4/assignment_loss‚0.1444661267367857 1loss/SRqqt2/detection_loss‚0.9574721351058947 1loss/SRqqt4/detection_loss‚2.3774526201188566 2loss/SRqqt1/assignment_loss‚1.3092147811108994«ÝÒ¡Y“Ÿ‚(:ˆ Ò°êÔ­‰–ÿ gpu.0.gpu‚54.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚165.598' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚41.3995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ƒgÆm ‚(:â Ò°êÔ¹½Ó€ disk.md0p1.out‚ 670.84375 disk.nvme0n1.in‚ 603.4453125 disk.sda.out‚696.28369140625 disk.rbd2.in‚0 disk.sda1.in‚0) disk./.usagePercent‚75.84096581879793 disk.dm-0.in‚ 1102.2421875 disk.dm-0.out‚ 71110.4296875# disk.nvme1n1.out‚39234.94921875 proc.cpu.threads‚17 disk.sdb1.out‚0 disk.rbd2.out‚ 1.63330078125# proc.memory.rssMB‚ 3002.45703125 disk.md0p1.in‚ 5.12890625 disk.sda2.in‚ 4.30078125 disk.nvme1n1.in‚ 498.796875$ memory_percent‚92.05732345581055 disk.sdb.out‚696.28369140625 disk.sdb1.in‚0 disk.sdb.in‚0.828125 disk.rbd0.in‚0 disk.md0.out‚ 670.84375 disk.sdb2.in‚0.828125 cpu‚25.213674677732012! disk.sda2.out‚696.28369140625! disk.sdb2.out‚696.28369140625 disk.sda1.out‚0' proc.memory.availableMB‚ 1301.328125 disk.md0.in‚ 5.12890625 disk.sda.in‚ 4.30078125 network.sent‚ 1.2827937e+07 network.recv‚ 1.95667e+06# disk.nvme0n1.out‚31875.48046875 disk.rbd0.out‚ 1.68359375) proc.memory.percent‚18.32554340362549% disk./.usageGB‚2712.4402923583984˜¯’¡‚(:‡ á°êÔϨþ gpu.0.gpu‚31.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚36.0 gpu.0.powerWatts‚114.408' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.602 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0– op¢‚(:å á°êÔ·øÿ network.recv‚ 1.961147e+06 disk.rbd0.out‚ 1.68359375 disk.dm-0.in‚ 1102.2421875 disk.sda.in‚ 4.30078125 proc.cpu.threads‚17! disk.nvme0n1.out‚ 31875.546875# disk.nvme1n1.out‚39393.83984375 disk.sdb1.out‚0 disk.md0.in‚ 5.12890625 disk.rbd2.in‚0 disk.rbd2.out‚ 1.63330078125 disk.md0p1.in‚ 5.12890625( proc.memory.availableMB‚ 1301.2734375 disk.sdb2.in‚0.828125# proc.memory.rssMB‚ 3002.45703125) proc.memory.percent‚18.32554340362549 disk.sdb2.out‚698.2763671875 disk.dm-0.out‚71269.38671875 disk.nvme0n1.in‚ 603.4453125 disk.sdb.in‚0.828125 disk.rbd0.in‚0 disk.sda1.out‚0 disk.sdb.out‚698.2763671875 disk.sdb1.in‚0 disk.md0.out‚ 672.78515625 cpu‚25.213196498144974 network.sent‚ 1.2934915e+07 disk.sda2.in‚ 4.30078125 disk.sda2.out‚698.2763671875 disk.nvme1n1.in‚ 498.796875 disk.sda1.in‚0 disk.md0p1.out‚ 672.78515625$ memory_percent‚92.05765724182129) disk./.usagePercent‚75.84483684949022% disk./.usageGB‚2712.5787391662598 disk.sda.out‚698.27636718751O;²o£‚(:ä ð°êÔâÖúþ disk.dm-0.in‚ 1102.2421875 disk.dm-0.out‚ 71385.6953125 disk.rbd0.out‚ 1.68359375 disk.sdb1.in‚0 disk.sda1.in‚0) proc.memory.percent‚18.32554340362549 disk.sda.out‚699.7724609375 disk.rbd0.in‚0 disk.md0.in‚ 5.12890625 disk.rbd2.out‚ 1.63330078125 network.sent‚ 1.294085e+07 disk.nvme1n1.in‚ 498.796875$ disk./.usageGB‚2712.582588195801 disk.md0p1.out‚ 674.20703125 disk.sda2.in‚ 4.30078125# disk.nvme1n1.out‚39510.01953125 disk.sdb1.out‚0 disk.sdb2.out‚699.7724609375 disk.sda1.out‚0$ memory_percent‚92.05765724182129 disk.sdb.out‚699.7724609375 disk.sdb2.in‚0.828125 proc.cpu.threads‚17 disk.md0p1.in‚ 5.12890625 disk.nvme0n1.in‚ 603.4453125 disk.sdb.in‚0.828125# proc.memory.rssMB‚ 3002.45703125 network.recv‚ 1.965692e+06) disk./.usagePercent‚75.84494446996712 disk.sda.in‚ 4.30078125 disk.sda2.out‚699.7724609375 disk.rbd2.in‚0( proc.memory.availableMB‚ 1301.2734375# disk.nvme0n1.out‚31875.67578125 disk.md0.out‚ 674.20703125 cpu‚25.212763742350393øØÃ(“¤‚(:ˆ ð°êÔÂåÑ‘ gpu.0.gpu‚35.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚176.738' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚44.1845 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0çÏè?”¥‚(:‰ ÿ°êÔª‰ßý gpu.0.gpu‚40.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚173.059' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.26475 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0NJŒf¦‚(:Û ÿ°êÔ¨»Íþ disk.md0p1.out‚ 675.83984375 disk.rbd0.in‚0 disk.rbd0.out‚ 1.68359375 disk.sdb1.in‚0 disk.sdb2.out‚ 701.451171875 disk.dm-0.in‚ 1102.2421875 disk.dm-0.out‚ 71477.3671875 disk.md0p1.in‚ 5.12890625 disk.sdb1.out‚0 disk.rbd2.out‚ 1.63330078125" disk.nvme0n1.out‚ 31875.6796875) proc.memory.percent‚18.32554340362549 cpu‚25.212286973649306 disk.sda.in‚ 4.30078125 disk.sda2.out‚ 701.451171875 disk.sdb2.in‚0.828125% disk./.usageGB‚2712.3726387023926 disk.rbd2.in‚0 disk.sda1.out‚0& proc.memory.availableMB‚ 1301.40625# proc.memory.rssMB‚ 3002.45703125 disk.nvme0n1.in‚ 603.4453125) disk./.usagePercent‚75.83907419426075 disk.sda1.in‚0 disk.sdb.in‚0.828125 proc.cpu.threads‚17 network.sent‚ 1.2946827e+07 network.recv‚ 1.969714e+06 disk.nvme1n1.in‚ 498.796875 disk.nvme1n1.out‚ 39601.6875 disk.md0.out‚ 675.83984375 disk.sda.out‚ 701.451171875 disk.sda2.in‚ 4.30078125 disk.md0.in‚ 5.12890625 disk.sdb.out‚ 701.451171875$ memory_percent‚92.05684661865234ÑþØ2§- _wandbruntime‚6111  _runtime‚6111,[‚QǨÁ trainer/global_step‚5249 % _timestamp‚1.7885164862404413e+09 _runtime‚6111.710817407 _step‚428 & lr-AdamW/pg1‚0.0005975715957110487 & lr-AdamW/pg2‚0.0005975715957110487IzôËÌ©Æ & lr-AdamW/pg2‚0.0005975715957110487 trainer/global_step‚5249 % _timestamp‚1.7885164862404413e+09 _runtime‚6111.710817407 _step‚428 & lr-AdamW/pg1‚0.0005975715957110487¬S B¦2ª- _wandbruntime‚6111  _runtime‚6111ÄÀ¹'ú'«ôO F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450780 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚437550975490048 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚31518269 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 trainer/global_step‚5249 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 131689307 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 131663219 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6230161757696 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _step‚429 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 108324708 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚489149410068992 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6229934994944 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚421959116893168 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955591168 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177047040 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 250702848 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚67358933 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 108324706 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 131689307 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 108324708 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26088 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚427903431426798 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5945211945050 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411420 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917410816 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5967457913344 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5967457913344 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26090 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚431584416984576 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 108324706 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚67358935 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 226762752 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚431584416984576 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407872 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚437551892900864 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚489149660771840 % _timestamp‚1.7885164862415102e+09 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 398533632 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 131663219 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 240014015 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚482919475074048 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5944314533630 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007241 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407872 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚431584434987520 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚437550975490048 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5966558505472 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1212 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 _runtime‚6111.711972761 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410816 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚608 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚427904345877578 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚437551892900864 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚98877204 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚606 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚421959133932528 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712ÓÇÿ'¬ùO WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 240014015 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚482919475074048 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5944314533630 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007241 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407872 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚431584434987520 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚437550975490048 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5966558505472 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1212 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 _runtime‚6111.711972761 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410816 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚608 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚427904345877578 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚437551892900864 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚98877204 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚606 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚421959133932528 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450780 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚437550975490048 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚31518269 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 trainer/global_step‚5249 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 131689307 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 131663219 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6230161757696 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _step‚429 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 108324708 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚489149410068992 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6229934994944 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚421959116893168 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955591168 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177047040 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 250702848 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚67358933 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 108324706 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 131689307 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 108324708 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26088 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚427903431426798 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5945211945050 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411420 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917410816 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5967457913344 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5967457913344 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26090 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚431584416984576 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 108324706 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚67358935 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 226762752 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚431584416984576 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407872 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚437551892900864 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚489149660771840 % _timestamp‚1.7885164862415102e+09 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 398533632 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 131663219­Ø׈Š2­- _wandbruntime‚6112  _runtime‚6112ª³¿ Ç ®þM =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚427983434633232 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 108344604 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31686 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6231324409344 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6231101854208 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007241 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5967671308800 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚98894565 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚615 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 131713924 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 240058530 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 246495232 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917424128 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚483008794294784 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450988 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚422037114026656 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚431664047280128 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 131713924 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26114 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚489240142644224 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 240058530 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚437632636013056 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899421184 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚31523869 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177047040 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚31523256 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 398533632 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 240032414 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 222555136 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚6112.668283743 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 108344606 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚613 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 108344604 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚67371311 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚483008818234880 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 trainer/global_step‚5249 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411628 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5945423194948 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚98895180 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5967671308800 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚437632636013056 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚67371309 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1187 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984ùƒæ! '¯ƒN H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 240032414 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 222555136 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚6112.668283743 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 108344606 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚613 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 108344604 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚67371311 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚483008818234880 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 trainer/global_step‚5249 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411628 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5945423194948 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚98895180 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5967671308800 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚437632636013056 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚67371309 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1187 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚427983434633232 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 108344604 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31686 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6231324409344 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6231101854208 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007241 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5967671308800 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚98894565 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚615 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 131713924 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 240058530 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 246495232 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917424128 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚483008794294784 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450988 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚422037114026656 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚431664047280128 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 131713924 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26114 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚489240142644224 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 240058530 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚437632636013056 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899421184 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚31523869 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177047040 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚31523256 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 398533632 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728®'ÿ{l2°- _wandbruntime‚6112  _runtime‚6112iEž˜X±Ò 1loss/FBt4/assignment_loss‚0.24166022127853162 /loss/SRqqt4/detection_loss‚2.93679420898358 epoch‚4 2loss/SRqqt1/assignment_loss‚1.2077450415651714 2loss/SRqqt3/assignment_loss‚1.2387565164005054 2loss/SRqqt4/assignment_loss‚1.7080156456989546 1loss/FBt3/assignment_loss‚0.46405281563000667 .loss/FRt4/assignment_loss‚3.09850134070103 1loss/FBt2/assignment_loss‚0.12924233015515055 /loss/FRt2/detection_loss‚1.5860668546995338 /loss/FRt4/detection_loss‚3.0864379428900204 1loss/SRqqt1/detection_loss‚1.4708486925810575 1loss/SRqqt2/detection_loss‚1.2846933024726352 .loss/FBt3/detection_loss‚2.239874793932988 2loss/SRqqt2/assignment_loss‚0.7603960892527687 0loss/FBt1/assignment_loss‚0.4400264825177146 .loss/FBt1/detection_loss‚2.177351123094559 /loss/FBt4/detection_loss‚2.3681150608592563 %loss/total_loss‚38.63186619915312 .loss/FRt3/detection_loss‚2.268952262543497 % _timestamp‚1.7885164871992054e+09 /loss/FRt2/assignment_loss‚0.865283232562396 .loss/FRt1/detection_loss‚1.990027236008254 1loss/SRqqt3/detection_loss‚1.6651770605468283 /loss/FBt2/detection_loss‚0.5548581360425386 0loss/FRt3/assignment_loss‚2.1679084428719113 trainer/global_step‚5249 _runtime‚6112.669336098 _step‚431 /loss/FRt1/assignment_loss‚2.681081365864232ÚÉ a]²× .loss/FBt3/detection_loss‚2.239874793932988 2loss/SRqqt2/assignment_loss‚0.7603960892527687 0loss/FBt1/assignment_loss‚0.4400264825177146 .loss/FBt1/detection_loss‚2.177351123094559 /loss/FBt4/detection_loss‚2.3681150608592563 %loss/total_loss‚38.63186619915312 .loss/FRt3/detection_loss‚2.268952262543497 % _timestamp‚1.7885164871992054e+09 /loss/FRt2/assignment_loss‚0.865283232562396 .loss/FRt1/detection_loss‚1.990027236008254 1loss/SRqqt3/detection_loss‚1.6651770605468283 /loss/FBt2/detection_loss‚0.5548581360425386 0loss/FRt3/assignment_loss‚2.1679084428719113 trainer/global_step‚5249 _runtime‚6112.669336098 _step‚431 /loss/FRt1/assignment_loss‚2.681081365864232 1loss/FBt4/assignment_loss‚0.24166022127853162 /loss/SRqqt4/detection_loss‚2.93679420898358 epoch‚4 2loss/SRqqt1/assignment_loss‚1.2077450415651714 2loss/SRqqt3/assignment_loss‚1.2387565164005054 2loss/SRqqt4/assignment_loss‚1.7080156456989546 1loss/FBt3/assignment_loss‚0.46405281563000667 .loss/FRt4/assignment_loss‚3.09850134070103 1loss/FBt2/assignment_loss‚0.12924233015515055 /loss/FRt2/detection_loss‚1.5860668546995338 /loss/FRt4/detection_loss‚3.0864379428900204 1loss/SRqqt1/detection_loss‚1.4708486925810575 1loss/SRqqt2/detection_loss‚1.2846933024726352¯Xu)”³‚(:‰ Ž±êÔš€þ gpu.0.gpu‚48.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚170.559' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.63975 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¬sð†v´‚(:ë Ž±êÔ¯”û€ disk.sdb.in‚0.828125 disk.sda.in‚ 4.30078125 disk.sda2.in‚ 4.30078125! disk.sda2.out‚703.16748046875) proc.memory.availableMB‚ 1301.24609375 network.recv‚ 1.974624e+06) disk./.usagePercent‚75.84324302115873% disk./.usageGB‚2712.5217361450195 disk.sdb1.out‚0 disk.md0.out‚ 677.48828125! disk.sdb2.out‚703.16748046875 disk.sda1.out‚0$ memory_percent‚92.05782413482666" disk.nvme0n1.out‚ 31876.0859375 disk.sda.out‚703.16748046875 disk.sda1.in‚0 disk.md0p1.out‚ 677.48828125 disk.nvme0n1.in‚ 603.4453125 network.sent‚ 1.2953106e+07 disk.sdb1.in‚0 disk.rbd0.out‚ 1.68359375# proc.memory.rssMB‚ 3002.45703125 cpu‚25.211829973087116 disk.nvme1n1.in‚ 498.796875 disk.rbd2.out‚ 1.63330078125 disk.rbd0.in‚0) proc.memory.percent‚18.32554340362549# disk.nvme1n1.out‚39702.44140625 disk.sdb2.in‚0.828125 disk.dm-0.out‚71578.52734375 disk.sdb.out‚703.16748046875 disk.md0.in‚ 5.12890625 disk.rbd2.in‚0 disk.dm-0.in‚ 1102.2421875 proc.cpu.threads‚17 disk.md0p1.in‚ 5.12890625zÞ w”µ‚(:‰ ±êÔîé”ÿ gpu.0.gpu‚31.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚116.583' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚29.14575 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0=©•DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚441588144847872 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6022213085696 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚487385460050944 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411420 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚441589062258688 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410816 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚435565949765120 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 248605696 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _step‚433 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6022213085696 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚435565931762176 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407872 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6288222740992 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 132896121 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚431858797241086 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1212 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚493673658851840 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26088 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 trainer/global_step‚5299 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚99782745 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 398533632 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚441588144847872 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2î=ú×þ'¾øO TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚31805012 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚493673907457536 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955591168 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407872 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚493673658851840 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26088 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 trainer/global_step‚5299 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚99782745 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 398533632 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚441588144847872 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214665 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 242215727 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 132922209 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 109319606 _runtime‚6164.473900298 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007241 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚435565949765120 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 242241817 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚594 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚67977733 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚31805606 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 109319608 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177047040 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 109319608 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚596 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26088 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6023112493568 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26090 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6288447406592 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 224665600 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚431857882790306 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6000658602158 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚441588144847872 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6022213085696 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚487385460050944 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411420 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚441589062258688 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410816 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚435565949765120 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 248605696 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _step‚433 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6022213085696 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚435565931762176 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407872 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6288222740992 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 132896121 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚431858797241086 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1212 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 trainer/global_step‚5299 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 132920712 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1187 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 109339506 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 109339506 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6001767263684 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917424128 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6023325889024 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 207875072 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411628 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007241 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 242286332 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚31810709 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚585 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 398533632 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚67990111 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26114 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚67990109 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚435645580060672 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 242260216 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 132946826 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6289618329088 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚425936118733056 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 242286332 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚99800818 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚425936101693696 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚99801403 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450988 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚431936971545752 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚583 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 trainer/global_step‚5299 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 132920712 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1187 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 109339506 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 109339506 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6001767263684 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917424128 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6023325889024 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 207875072 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411628 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007241 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 242286332 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚31810709 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚585 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 398533632 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚67990111 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26114 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚67990109 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚435645580060672 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 242260216 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 132946826 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6289618329088 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚425936118733056 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 242286332 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚99800818 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚425936101693696 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚99801403 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450988 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚431936971545752 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚583 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 EDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚445625314205696 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚631 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 244411392 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 110314506 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚32093207 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6346286215680 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 _runtime‚6217.509697837 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26090 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚68596533 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 398533632 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚491851397147648 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 100689111 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚439547464542720 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 110314508 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26088 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 134155111 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6078767073792 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1212 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 110314508 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚498197927774720 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 134129023 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6077867665920 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 110314506 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚491851421087744 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450780 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 trainer/global_step‚5349 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 134129023 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177047040 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚439547446539776 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚429757143345328 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411420 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26088 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚445626231616512 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚32092578 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚445625314205696 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚631 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 244411392 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 110314506 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚32093207 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6346286215680 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 _runtime‚6217.509697837 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26090 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚68596533 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 398533632 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚491851397147648 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 100689111 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚439547464542720 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 110314508 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26088 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 134155111 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6078767073792 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1212 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 110314508 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚498197927774720 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 134129023 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6077867665920 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 110314506 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚491851421087744 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450780 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 trainer/global_step‚5349 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 134129023 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177047040 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚439547446539776 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚429757143345328 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411420 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26088 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚445626231616512 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚32092578 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 JDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6079879890432 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 134179728 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚68608911 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1187 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚491940716368384 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚439627094838272 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6078980469248 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177047040 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚637 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚6218.565285715 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚498288157070336 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450988 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 134153614 REDeviceStatsMonitor.on_traéi “in_batch_end/active_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177047040 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 100707747 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6347440701952 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 100707108 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚32098199 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 224652288 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 244514134 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411628 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 trainer/global_step‚5349 % _timestamp‚1.7885165930941772e+09 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚32098836 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899421184 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6057213920792 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007241 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 _step‚438 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚429835123439456 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚435891422909260 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 134179728 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚429835106400096 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 110334406 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚445706974728704 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚639 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚491940740308480 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚439627076835328 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 110334404 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 244514134 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411628 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 trainer/global_step‚5349 % _timestamp‚1.7885165930941772e+09 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚32098836 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899421184 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6057213920792 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007241 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 _step‚438 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚429835123439456 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚435891422909260 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 134179728 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚429835106400096 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 110334406 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚445706974728704 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚639 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚491940740308480 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚439627076835328 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 110334404 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6079879890432 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 134179728 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚68608911 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1187 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚491940716368384 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚439627094838272 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6078980469248 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177047040 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚637 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚6218.565285715 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚498288157070336 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450988 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 134153614 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177047040 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 100707747 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6347440701952 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 100707108 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚32098199 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 224652288 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0¶zŽÕ2Ö- _wandbruntime‚6218  _runtime‚6218ÑåW×Ñ % _timestamp‚1.7885165930966647e+09 /loss/FBt4/detection_loss‚3.0985368215120754 trainer/global_step‚5349 /loss/FRt1/assignment_loss‚3.271059540686784 2loss/SRqqt1/assignment_loss‚1.3043812305519455 0loss/FBt3/assignment_loss‚0.3275635537534072 _step‚439 0loss/FRt2/assignment_loss‚1.4503472085092581 1loss/FBt1/assignment_loss‚0.10187815032550133 -loss/FRt3/detection_loss‚2.08376629376898 2loss/SRqqt3/assignment_loss‚1.4053849764337594 /loss/FBt3/detection_loss‚3.0387088134884834 epoch‚4 1loss/SRqqt2/detection_loss‚0.9210756279528141 .loss/FBt1/detection_loss‚2.985567605495453 1loss/SRqqt4/assignment_loss‚1.460215072269025 /loss/FRt2/detection_loss‚1.9849985290830041 0loss/SRqqt1/detection_loss‚1.776398510525101 1loss/SRqqt3/detection_loss‚1.5255172414197162 /loss/FBt2/detection_loss‚0.7660050576428572 _runtime‚6218.566861212 /loss/FRt3/assignment_loss‚2.412763301207095 .loss/FRt4/assignment_loss‚4.06069743235906 2loss/SRqqt2/assignment_loss‚0.8941273055305439 /loss/FRt1/detection_loss‚1.9994343070869887 .loss/FRt4/detection_loss‚2.970133650302887 1loss/FBt4/assignment_loss‚0.16189727677892035 1loss/SRqqt4/detection_loss‚2.9173808331074924 %loss/total_loss‚43.04528116494009 0loss/FBt2/assignment_loss‚0.1274428251489356Ûñv\ØÖ 2loss/SRqqt2/assignment_loss‚0.8941273055305439 /loss/FRt1/detection_loss‚1.9994343070869887 .loss/FRt4/detection_loss‚2.970133650302887 1loss/FBt4/assignment_loss‚0.16189727677892035 1loss/SRqqt4/detection_loss‚2.9173808331074924 %loss/total_loss‚43.04528116494009 0loss/FBt2/assignment_loss‚0.1274428251489356 % _timestamp‚1.7885165930966647e+09 /loss/FBt4/detection_loss‚3.0985368215120754 trainer/global_step‚5349 /loss/FRt1/assignment_loss‚3.271059540686784 2loss/SRqqt1/assignment_loss‚1.3043812305519455 0loss/FBt3/assignment_loss‚0.3275635537534072 _step‚439 0loss/FRt2/assignment_loss‚1.4503472085092581 1loss/FBt1/assignment_loss‚0.10187815032550133 -loss/FRt3/detection_loss‚2.08376629376898 2loss/SRqqt3/assignment_loss‚1.4053849764337594 /loss/FBt3/detection_loss‚3.0387088134884834 epoch‚4 1loss/SRqqt2/detection_loss‚0.9210756279528141 .loss/FBt1/detection_loss‚2.985567605495453 1loss/SRqqt4/assignment_loss‚1.460215072269025 /loss/FRt2/detection_loss‚1.9849985290830041 0loss/SRqqt1/detection_loss‚1.776398510525101 1loss/SRqqt3/detection_loss‚1.5255172414197162 /loss/FBt2/detection_loss‚0.7660050576428572 _runtime‚6218.566861212 /loss/FRt3/assignment_loss‚2.412763301207095 .loss/FRt4/assignment_loss‚4.06069743235906·4g`ð”Ù‚(:‰ ÷±êԀ䞀 gpu.0.gpu‚26.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚37.0 gpu.0.powerWatts‚114.131' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.53275 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ðºlÚ‚(:á ÷±êÔ‰ô©€ disk.sda.in‚ 4.30078125 disk.sdb1.out‚0 disk.rbd0.out‚ 1.68359375 disk.dm-0.out‚ 72569.7421875 disk.sdb2.in‚0.828125 disk.sda1.in‚0 disk.sda.out‚715.39111328125 disk.md0.in‚ 5.12890625 cpu‚25.208836724345026 network.sent‚ 1.3290095e+07 disk.sdb.out‚715.39111328125! disk.sda2.out‚715.39111328125# disk.nvme1n1.out‚40692.62890625 disk.sda1.out‚0 network.recv‚ 2.005673e+06! disk.sdb2.out‚715.39111328125 disk.nvme0n1.in‚ 603.4453125 disk.sdb.in‚0.828125) proc.memory.percent‚18.32554340362549 disk.md0p1.out‚689.25# disk.nvme0n1.out‚31877.11328125 disk.sdb1.in‚0 disk.md0.out‚689.25 disk.sda2.in‚ 4.30078125 disk.dm-0.in‚ 1102.24609375) disk./.usagePercent‚75.85106657118985$ disk./.usageGB‚2712.801544189453 disk.rbd2.out‚ 1.78271484375 disk.md0p1.in‚ 5.12890625 disk.nvme1n1.in‚ 498.80078125) proc.memory.availableMB‚ 1301.05859375 disk.rbd2.in‚0 disk.rbd0.in‚0$ memory_percent‚92.05896854400635# proc.memory.rssMB‚ 3002.45703125 proc.cpu.threads‚17«L’Û‚(:‡ †²êÔµõÕÿ gpu.0.gpu‚39.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚163.09' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚40.7725 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0½—uBwÜ‚(:ì †²êÔËòÿ! disk.sdb2.out‚717.22314453125 disk.sda1.out‚0# disk.nvme0n1.out‚31937.84765625 disk.sdb.out‚717.22314453125 proc.cpu.threads‚17 network.sent‚ 1.3387441e+07 disk.sdb1.out‚0 disk.sdb2.in‚0.828125 disk.md0p1.in‚ 5.12890625 disk.rbd0.out‚ 1.68359375% disk./.usageGB‚2712.8640060424805 disk.nvme1n1.in‚ 498.80078125 disk.sdb1.in‚0 disk.nvme0n1.in‚ 603.4453125 disk.dm-0.out‚ 72830.828125 disk.sda.in‚ 4.30078125 disk.sda2.in‚ 4.30078125) disk./.usagePercent‚75.85281303074281) proc.memory.percent‚18.32554340362549 network.recv‚ 2.011535e+06# disk.nvme1n1.out‚40892.98046875 disk.md0p1.out‚ 691.01953125 disk.sdb.in‚0.828125 disk.rbd0.in‚0 disk.sda1.in‚0 disk.sda.out‚717.22314453125$ memory_percent‚92.05758571624756 cpu‚25.20838033925941 disk.md0.in‚ 5.12890625 disk.md0.out‚ 691.01953125 disk.rbd2.out‚ 1.78271484375 disk.dm-0.in‚ 1102.24609375! disk.sda2.out‚717.22314453125) proc.memory.availableMB‚ 1301.28515625# proc.memory.rssMB‚ 3002.45703125 disk.rbd2.in‚0¨‚^=”Ý‚(:‰ •²êÔõšþ gpu.0.gpu‚34.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚37.0 gpu.0.powerWatts‚119.701' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚29.92525 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0òÊ? uÞ‚(:ê •²êԞ݅ÿ disk.sda1.out‚0 disk.dm-0.in‚ 1102.24609375) proc.memory.percent‚18.32554340362549 cpu‚25.20796941983951# proc.memory.rssMB‚ 3002.45703125) disk./.usagePercent‚75.84341495793649 disk.sda.out‚718.57080078125 disk.sdb1.in‚0 disk.nvme0n1.in‚ 603.4453125# disk.nvme0n1.out‚32006.23828125$ memory_percent‚92.05853939056396 disk.md0p1.out‚ 692.32421875! disk.sda2.out‚718.57080078125 disk.rbd0.in‚0! disk.sdb2.out‚718.57080078125 network.recv‚ 2.015294e+06 disk.nvme1n1.in‚ 498.80078125 disk.sdb1.out‚0 disk.md0.out‚ 692.32421875 disk.rbd2.out‚ 1.845703125 proc.cpu.threads‚17 network.sent‚ 1.3390638e+07 disk.rbd0.out‚ 1.68359375! disk.nvme1n1.out‚ 41017.828125 disk.rbd2.in‚0 disk.md0p1.in‚ 5.12890625 disk.dm-0.out‚73024.06640625 disk.sdb.out‚718.57080078125% disk./.usageGB‚2712.5278854370117 disk.sda.in‚ 4.30078125 disk.md0.in‚ 5.12890625 disk.sdb2.in‚0.828125 disk.sda1.in‚0 disk.sdb.in‚0.828125) proc.memory.availableMB‚ 1301.12890625 disk.sda2.in‚ 4.30078125‚B­u”ß‚(:‰ ¤²êÔá΋ÿ gpu.0.gpu‚52.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚156.911' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚39.22775 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Å?rà‚(:ç ¤²êÔ×Øûÿ disk.dm-0.in‚ 1102.24609375 disk.md0p1.in‚ 5.12890625 disk.md0p1.out‚ 693.87890625 disk.sda2.out‚720.1865234375 disk.sdb1.out‚0 disk.md0.in‚ 5.12890625 disk.sdb.in‚0.828125 disk.rbd2.out‚ 1.845703125$ memory_percent‚92.05706119537354 disk.sdb.out‚720.1865234375 disk.rbd0.in‚0 disk.sda1.in‚0# proc.memory.rssMB‚ 3002.45703125# disk.nvme0n1.out‚32161.91796875 disk.sda.in‚ 4.30078125 disk.sda2.in‚ 4.30078125! disk.nvme1n1.out‚ 41101.828125 disk.sdb2.in‚0.828125) proc.memory.availableMB‚ 1301.37109375% disk./.usageGB‚2712.5269317626953 disk.nvme0n1.in‚ 603.4453125 disk.nvme1n1.in‚ 498.80078125 disk.sdb1.in‚0 proc.cpu.threads‚17 network.sent‚ 1.3398862e+07 disk.dm-0.out‚73263.74609375 disk.rbd2.in‚0 disk.sda1.out‚0 cpu‚25.207505800237186 network.recv‚ 2.019551e+06) disk./.usagePercent‚75.84338829280347 disk.md0.out‚ 693.87890625 disk.sdb2.out‚720.1865234375) proc.memory.percent‚18.32554340362549 disk.sda.out‚720.1865234375 disk.rbd0.out‚ 1.68359375‡ÑiË2á- _wandbruntime‚6270  _runtime‚6270*ÌÇâÁ trainer/global_step‚5399 % _timestamp‚1.7885166448674865e+09 _runtime‚6270.337956389 _step‚440 & lr-AdamW/pg1‚0.0005803436910051367 & lr-AdamW/pg2‚0.0005803436910051367~'ëÌãÆ & lr-AdamW/pg2‚0.0005803436910051367 trainer/global_step‚5399 % _timestamp‚1.7885166448674865e+09 _runtime‚6270.337956389 _step‚440 & lr-AdamW/pg1‚0.0005803436910051367¸HÂ2ä- _wandbruntime‚6270  _runtime‚6270×ÏáTðåõO P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955591168 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 252800000 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚502721627084800 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚439766785517322 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407872 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007241 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917410816 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6133522246144 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚433656131012368 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 101593823 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚69215335 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚69215333 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 246671331 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚433656148051728 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 trainer/global_step‚5399 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚32379123 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚32378490 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚449662483563520 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚443528979320320 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6111551916374 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 _step‚441 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 111309408 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚443528961317376 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450780 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26090 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 135388013 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚443528979320320 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26088 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 111309408 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚633 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚496317358184448 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171¨Ëé(æúO RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 $ _timestamp‚1.788516644868668e+09 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007241 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917410816 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6133522246144 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚433656131012368 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 101593823 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚69215335 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚69215333 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 246671331 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚433656148051728 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 trainer/global_step‚5399 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚32379123 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚32378490 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚449662483563520 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚443528979320320 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6111551916374 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 _step‚441 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 111309408 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚443528961317376 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450780 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26090 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 135388013 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚443528979320320 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26088 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 111309408 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚633 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚496317358184448 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955591168 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 252800000 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚502721627084800 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚439766785517322 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407872 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 _runtime‚6271.308872738 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 111329304 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007241 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚449744144086528 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚443608591612928 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6134635049472 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177047040 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚502812347581952 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 246715820 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 111329304 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26114 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917424128 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 trainer/global_step‚5399 _step‚442 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚69227709 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955591168 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 246741936 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 228846592 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚433734128145856 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚32384268 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 252786688 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 % _timestamp‚1.7885166458380249e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚439845874272768 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚433734111106496 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚32384878 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 246715820 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 135412630 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411628 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚496406677405184 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚449744144086528 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6135534470656 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚439846788723756 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 111329306 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚443608591612928 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6112660577900 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 398533632 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6405646236672 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177047040 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚496406701345280 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899421184 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6405417390080 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6135534470656 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450988 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚443608609615872 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6134635049472 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 135386516 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚612 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26114 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 101611977 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 246741936 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 111329306 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 135412630 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚610 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚443608609615872 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6111763166272 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899421184 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 135386516 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1187 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 101611977 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 246741936 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 111329306 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 135412630 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚610 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚443608609615872 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6111763166272 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899421184 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 135386516 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1187 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 _runtime‚6271.308872738 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 111329304 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007241 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚449744144086528 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚443608591612928 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6134635049472 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177047040 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚502812347581952 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 246715820 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 111329304 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26114 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917424128 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 trainer/global_step‚5399 _step‚442 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚69227709 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955591168 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 246741936 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 228846592 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚433734128145856 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚32384268 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_byte½q s.all.current‚ 252786688 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 % _timestamp‚1.7885166458380249e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚439845874272768 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚433734111106496 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚32384878 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 246715820 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 135412630 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411628 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚496406677405184 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚449744144086528 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6135534470656 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚439846788723756 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 111329306 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚443608591612928 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6112660577900 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 398533632 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6405646236672 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177047040º=Àw22ê- _wandbruntime‚6271  _runtime‚6271”ÒŸw`ëÚ 1loss/SRqqt4/assignment_loss‚1.844776635368665 1loss/FBt2/assignment_loss‚0.14631270489037407 /loss/FRt1/detection_loss‚2.2423063365222045 /loss/FRt2/detection_loss‚1.3410533736547772 1loss/SRqqt1/detection_loss‚1.9431038395297238 trainer/global_step‚5399 0loss/FRt2/assignment_loss‚1.2731967388458971 0loss/FRt3/assignment_loss‚2.2769598557103063 1loss/SRqqt1/assignment_loss‚1.283237044256544 1loss/SRqqt2/detection_loss‚0.8974684095149165 /loss/FBt4/detection_loss‚2.9531011790037156 _step‚443 1loss/FBt1/assignment_loss‚0.03352901950314514 1loss/FBt3/assignment_loss‚0.20902492680943396 &loss/total_loss‚40.901692701779595 0loss/FRt1/assignment_loss‚2.5805532205395583 2loss/SRqqt2/assignment_loss‚1.1014543733170339 /loss/FRt3/detection_loss‚2.8499371287442985 1loss/SRqqt3/detection_loss‚1.4389393830131139 /loss/FBt1/detection_loss‚2.4998751987110484 /loss/FRt4/detection_loss‚3.0798635887248174 /loss/FBt2/detection_loss‚0.8174292172208942 _runtime‚6271.309919353 0loss/FRt4/assignment_loss‚3.0189592710563113 1loss/FBt4/assignment_loss‚0.14112615065969294 0loss/SRqqt4/detection_loss‚2.822009132968055 .loss/FBt3/detection_loss‚2.686926782131195 % _timestamp‚1.7885166458397875e+09 2loss/SRqqt3/assignment_loss‚1.4205491910838792 epoch‚4¦¾¯eìß 0loss/SRqqt4/detection_loss‚2.822009132968055 .loss/FBt3/detection_loss‚2.686926782131195 % _timestamp‚1.7885166458397875e+09 2loss/SRqqt3/assignment_loss‚1.4205491910838792 epoch‚4 1loss/SRqqt4/assignment_loss‚1.844776635368665 1loss/FBt2/assignment_loss‚0.14631270489037407 /loss/FRt1/detection_loss‚2.2423063365222045 /loss/FRt2/detection_loss‚1.3410533736547772 1loss/SRqqt1/detection_loss‚1.9431038395297238 trainer/global_step‚5399 0loss/FRt2/assignment_loss‚1.2731967388458971 0loss/FRt3/assignment_loss‚2.2769598557103063 1loss/SRqqt1/assignment_loss‚1.283237044256544 1loss/SRqqt2/detection_loss‚0.8974684095149165 /loss/FBt4/detection_loss‚2.9531011790037156 _step‚443 1loss/FBt1/assignment_loss‚0.03352901950314514 1loss/FBt3/assignment_loss‚0.20902492680943396 &loss/total_loss‚40.901692701779595 0loss/FRt1/assignment_loss‚2.5805532205395583 2loss/SRqqt2/assignment_loss‚1.1014543733170339 /loss/FRt3/detection_loss‚2.8499371287442985 1loss/SRqqt3/detection_loss‚1.4389393830131139 /loss/FBt1/detection_loss‚2.4998751987110484 /loss/FRt4/detection_loss‚3.0798635887248174 /loss/FBt2/detection_loss‚0.8174292172208942 _runtime‚6271.309919353 0loss/FRt4/assignment_loss‚3.0189592710563113 1loss/FBt4/assignment_loss‚0.14112615065969294»SäFÊõÄ trainer/global_step‚5449 % _timestamp‚1.7885166973502629e+09 _runtime‚6322.820642378 _step‚444 % lr-AdamW/pg1‚0.000574536311495621 % lr-AdamW/pg2‚0.000574536311495621¼6¦Â‹2ö- _wandbruntime‚6322  _runtime‚6322ñ=t;ü'÷öO YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚447510476094976 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 398533632 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26088 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚500783343161344 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 136620915 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 248605696 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚32664687 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 _runtime‚6322.821814173 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚69834135 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚437555152758128 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 136594827 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚453699652921344 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 224665600 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007241 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 248925223 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚453699652921344 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177047040 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955591168 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚607 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 248899133 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 112304308 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚447510494097920 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1187 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚507246011922944 % _timestamp‚1.7885166973513484e+09 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955591168 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 136594827 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚32665294 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26088 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6190076234240 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917410816 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚443721236880830 _step‚445 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚453700570332160 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚453699652921344 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177047040 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955591168 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚607 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 248899133 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 112304308 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚447510494097920 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1187 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚507246011922944 % _timestamp‚1.7885166973513484e+09 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955591168 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 136594827 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚32665294 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26088 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6190076234240 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917410816 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚443721236880830 _step‚445 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚453700570332160 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007241 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 248925223 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 JDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26114 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177047040 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚32670396 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ ZW“â 1175007241 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 112324204 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 248943622 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚437633132852256 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 248969738 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955591168 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚447590124393472 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6463595871744 _step‚446 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 102516905 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 136619418 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚453781313444352 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚443801240087264 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6168107235008 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31686 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26114 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚500872662382080 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450988 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 136619418 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚604 % _timestamp‚1.7885166983034153e+09K™ 'û…N N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955591168 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚447590124393472 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6463595871744 _step‚446 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 102516905 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 136619418 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚453781313444352 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚443801240087264 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6168107235008 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31686 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26114 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚500872662382080 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450988 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 136619418 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚604 % _timestamp‚1.7885166983034153e+09 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 112324206 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 254883840 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 136645532 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚606 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26114 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177047040 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚32670396 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007241 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 112324204 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 248943622 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚437633132852256 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 248969738 FDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 137853817 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚457736822279168 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 _step‚449 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177047040 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6520732061696 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6245730814464 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 224665600 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚505249280258048 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚441454140425168 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214665 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚457737739689984 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 137827729 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917410816 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6244831406592 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955591168 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚451492008875520 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚601 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6244831406592 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚32951189 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411420 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚70452933 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 248605696 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26090 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚447676602695118 trainer/global_step‚5499 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 251126935 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007241 XKDeviceS(ì~tatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚32951790 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1187 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚441454157464528 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 113299208 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚505249304198144 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚451491990872576 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410816 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6245730814464 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955591168 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 113299206 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 113299208 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚451492008875520 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 103404725 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 103404122 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450780 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚457736822279168 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6222445230590 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6221547819170 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚603 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚70452935 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _runtime‚6374.272399616 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 113299206 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 251153025 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1212 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407872ñröè(ŠûO JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411420 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚70452933 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 248605696 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26090 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚447676602695118 trainer/global_step‚5499 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 251126935 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007241 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚32951790 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1187 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚441454157464528 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 113299208 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚505249304198144 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚451491990872576 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410816 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6245730814464 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955591168 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 113299206 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 113299208 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚451492008875520 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 103404725 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 103404122 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450780 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚457736822279168 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6222445230590 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6221547819170 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚603 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚70452935 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _runtime‚6374.272399616 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 113299206 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 251153025 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1212 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407872 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26088 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚457737739689984 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 137853817 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚457736822279168 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 _step‚449 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177047040 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6520732061696 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6245730814464 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 224665600 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚505249280258048 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚441454140425168 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214665 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚457737739689984 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 137827729 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917410816 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6244831406592 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955591168 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚451492008875520 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚601 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6244831406592 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚32951189 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984ÁŸ_V6’‹‚(:‡ ³êÔ¨°Áþ gpu.0.gpu‚55.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚171.168' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.792 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ûòúxnŒ‚(:ã ³êÔïè†ÿ disk.rbd2.out‚ 1.845703125 disk.dm-0.out‚ 74041.9609375 disk.md0p1.out‚ 705.3046875 disk.sda1.out‚0 disk.md0p1.in‚ 5.12890625" disk.nvme1n1.out‚ 41718.0703125! disk.sda2.out‚732.02490234375# memory_percent‚92.0616865158081% disk./.usageGB‚2713.0845260620117 disk.sdb.out‚732.02490234375 disk.md0.out‚ 705.3046875 disk.rbd0.out‚ 1.68359375 disk.rbd2.in‚0! disk.nvme0n1.out‚ 32323.890625) proc.memory.percent‚18.32554340362549 disk.sda1.in‚0 disk.nvme0n1.in‚ 603.4453125# proc.memory.rssMB‚ 3002.45703125 network.recv‚ 2.050165e+06 disk.sda2.in‚ 4.30078125 disk.rbd0.in‚0 disk.nvme1n1.in‚ 498.80078125 disk.sdb2.in‚0.828125! disk.sdb2.out‚732.02490234375 cpu‚25.2044999012677 network.sent‚ 1.3640508e+07( disk./.usagePercent‚75.8589788627837 disk.sda.in‚ 4.30078125 disk.dm-0.in‚ 1102.24609375 proc.cpu.threads‚17 disk.sdb.in‚0.828125 disk.sda.out‚732.02490234375 disk.sdb1.in‚0) proc.memory.availableMB‚ 1300.61328125 disk.sdb1.out‚0 disk.md0.in‚ 5.12890625! ¤¯2- _wandbruntime‚6375  _runtime‚6375ùŸÓ‚'Ž€N % _timestamp‚1.7885167497694192e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚451571621168128 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚505338623418880 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26114 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚511860512195072 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚441532137558656 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6246843631104 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚634 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 248592384 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 113319104 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1187 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 251197540 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6245944209920 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 113319104 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 137852320 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31686 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 103422829 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚451571621168128 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚447755691450772 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 251171424 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚447754776999784 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 113319106 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899421184 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26114 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177047040 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 113319106 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚70465309 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6222656480488 trainer/global_step‚5499 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚505338599478784 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚441532120519296 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6223553892116 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917424128 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177047040 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚636 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 251197540 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6521888776192 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 224652288 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 398533632 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 103422193 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411628 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚451571621168128 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚447755691450772 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 251171424 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚447754776999784 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 113319106 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899421184 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26114 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177047040 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 113319106 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚70465309 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6222656480488 trainer/global_step‚5499 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚505338599478784 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚441532120519296 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6223553892116 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917424128 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177047040 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚636 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 251197540 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6521888776192 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 224652288 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 398533632 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 103422193 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411628 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26114 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚511860512195072 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ^JDeviceStatsMonitor.on_train_batcÏéZ€ó h_end/requested_bytes.large_pool.allocated‚441532137558656 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6246843631104 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚634 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 248592384 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 113319104 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1187 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 251197540 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6245944209920 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 113319104 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 137852320 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31686 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 103422829 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 EDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚461773991636992 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚455473523653120 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 % _timestamp‚1.7885168004527242e+09 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚461774909047808 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 253380827 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚33237562 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚609 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 114294106 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214665 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 114294106 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚509715241294848 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚71071733 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6578647556096 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 114294108 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6300485986816 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚445353145131568 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917410816 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚509715265234944 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 trainer/global_step‚5549 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26090 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6277891887698 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚461773991636992 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26088 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚33236953 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚516293888850944 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 139060631 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6301385394688 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917410816 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚509715265234944 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 trainer/global_step‚5549 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26090 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6277891887698 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚461773991636992 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26088 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚33236953 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚516293888850944 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 139060631 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6301385394688 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚461773991636992 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚455473523653120 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 % _timestamp‚1.7885168004527242e+09 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚461774909047808 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 253380827 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚33237562 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚609 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 114294106 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214665 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 114294106 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚509715241294848 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚71071733 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6578647556096 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 114294108 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6300485986816 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚445353145131568 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931ÅîʸT2Ÿ- _wandbruntime‚6426  _runtime‚6426µaó} ÿM OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 114314006 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6302498211328 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚451709228363292 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6279000549224 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚454 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 114314004 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 253425342 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 114314004 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6579815280640 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚455553153948672 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6301598790144 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 139085222 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚461854734735872 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚445431142265056 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 104327374 trainer/global_step‚5549 _runtime‚6426.917334446 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚71084111 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 220457984 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007241 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚509804560515584 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26114 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26114 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6580035738624 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚451710142814280 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚445431125225696 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6278103137596 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6302498211328 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 104326774 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 $ _timestamp‚1.788516801446213e+09 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955591168 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917424128 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177047040 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 139111336 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚71084109 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚509804584455680 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 253399226 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177047040 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 114314006 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 139085222 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚455553153948672 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6301598790144 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31686 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚455553135945728 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚461855652160000 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899421184 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚516384620194304 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚455553135945728 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 139111336 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 398533632 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564k,1 '¡„N >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 253399226 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177047040 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 114314006 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 139085222 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚455553153948672 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6301598790144 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31686 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚455553135945728 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚461855652160000 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899421184 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚516384620194304 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚455553135945728 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 139111336 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 398533632 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 114314006 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6302498211328 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚451709228363292 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6279000549224 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚454 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 114314004 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 253425342 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 114314004 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6579815280640 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚455553153948672 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6301598790144 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 139085222 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚461854734735872 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚445431142265056 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 104327374 trainer/global_step‚5549 _runtime‚6426.917334446 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚71084111 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 220457984 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007241 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚509804560515584 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26114 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26114 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6580035738624 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚451710142814280 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚445431125225696 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6278103137596 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6302498211328 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 104326774 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 $ _timestamp‚1.788516801446213e+09 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955591168 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917424128 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177047040 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 139111336 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚71084109 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚509804584455680 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0ƪ\EX2¢- _wandbruntime‚6426  _runtime‚6426‰<ó.Y£Ó 2loss/SRqqt2/assignment_loss‚0.8623405631172331 2loss/SRqqt3/assignment_loss‚1.1792390765156597 2loss/SRqqt4/assignment_loss‚1.2720856594072807 1loss/SRqqt2/detection_loss‚0.9411339049469934 .loss/FBt1/detection_loss‚2.209884151816368 %loss/total_loss‚38.39558187284797 1loss/FBt1/assignment_loss‚0.19503518387950863 1loss/SRqqt3/detection_loss‚1.7890512530505658 -loss/FBt4/detection_loss‚2.46576703446252 /loss/FRt4/assignment_loss‚2.618033344254774 0loss/FBt3/assignment_loss‚0.5747113010535637 /loss/FRt4/detection_loss‚2.8818820118904114 /loss/FBt2/detection_loss‚0.5469317165680412 epoch‚4 trainer/global_step‚5549 _runtime‚6426.919042134 2loss/SRqqt1/assignment_loss‚1.1736970795318484 /loss/FRt1/detection_loss‚2.1937169760745885 0loss/SRqqt4/detection_loss‚2.118354161848893 .loss/FBt3/detection_loss‚2.351929763952891 % _timestamp‚1.7885168014488213e+09 _step‚455 /loss/FRt1/assignment_loss‚3.152227128927524 0loss/FRt3/assignment_loss‚3.1500654780134862 0loss/FBt2/assignment_loss‚0.1389458544452132 /loss/FRt2/detection_loss‚1.3804599885224251 0loss/FRt2/assignment_loss‚0.9765010350628903 0loss/FBt4/assignment_loss‚0.3339521046718801 /loss/FRt3/detection_loss‚2.2672628254002456 1loss/SRqqt1/detection_loss‚1.6223742754331656ùÓ ^¤Ø 0loss/FRt2/assignment_loss‚0.9765010350628903 0loss/FBt4/assignment_loss‚0.3339521046718801 /loss/FRt3/detection_loss‚2.2672628254002456 1loss/SRqqt1/detection_loss‚1.6223742754331656 2loss/SRqqt2/assignment_loss‚0.8623405631172331 2loss/SRqqt3/assignment_loss‚1.1792390765156597 2loss/SRqqt4/assignment_loss‚1.2720856594072807 1loss/SRqqt2/detection_loss‚0.9411339049469934 .loss/FBt1/detection_loss‚2.209884151816368 %loss/total_loss‚38.39558187284797 1loss/FBt1/assignment_loss‚0.19503518387950863 1loss/SRqqt3/detection_loss‚1.7890512530505658 -loss/FBt4/detection_loss‚2.46576703446252 /loss/FRt4/assignment_loss‚2.618033344254774 0loss/FBt3/assignment_loss‚0.5747113010535637 /loss/FRt4/detection_loss‚2.8818820118904114 /loss/FBt2/detection_loss‚0.5469317165680412 epoch‚4 trainer/global_step‚5549 _runtime‚6426.919042134 2loss/SRqqt1/assignment_loss‚1.1736970795318484 /loss/FRt1/detection_loss‚2.1937169760745885 0loss/SRqqt4/detection_loss‚2.118354161848893 .loss/FBt3/detection_loss‚2.351929763952891 % _timestamp‚1.7885168014488213e+09 _step‚455 /loss/FRt1/assignment_loss‚3.152227128927524 0loss/FRt3/assignment_loss‚3.1500654780134862 0loss/FBt2/assignment_loss‚0.1389458544452132 /loss/FRt2/detection_loss‚1.3804599885224251Çd Ù”¥‚(:‰ ɳêÔÂåþ gpu.0.gpu‚52.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚39.0 gpu.0.powerWatts‚176.461' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚44.11525 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0a„Ñn¦‚(:ã ɳêÔ”éºþ disk.sdb1.out‚0 proc.cpu.threads‚17 disk.sdb.in‚0.828125 disk.sdb.out‚739.4677734375 disk.nvme1n1.in‚ 498.80078125 disk.rbd2.out‚ 1.845703125 disk.sda.out‚739.4677734375$ memory_percent‚92.06230640411377 disk.rbd0.in‚0 disk.rbd0.out‚ 1.68359375 network.sent‚ 1.3845465e+07) disk./.usagePercent‚75.85196145305434! disk.nvme0n1.out‚ 32324.171875 disk.sda2.in‚ 4.30078125 disk.md0.in‚ 5.12890625 disk.rbd2.in‚0 disk.sda1.in‚0 disk.md0p1.in‚ 5.12890625 disk.nvme0n1.in‚ 603.4453125) proc.memory.availableMB‚ 1300.51171875# proc.memory.rssMB‚ 3002.45703125) proc.memory.percent‚18.32554340362549 network.recv‚ 2.067926e+06 disk.sda1.out‚0 disk.dm-0.out‚ 75043.5234375 disk.sda.in‚ 4.30078125 disk.sda2.out‚739.4677734375 disk.md0.out‚ 712.4921875 disk.sdb2.in‚0.828125 cpu‚25.202914918947382% disk./.usageGB‚2712.8335494995117 disk.dm-0.in‚ 1102.24609375 disk.md0p1.out‚ 712.4921875 disk.sdb1.in‚0 disk.sdb2.out‚739.4677734375" disk.nvme1n1.out‚ 42719.3515625¼Û%¬’§‚(:‡ سêÔÎýÂÿ gpu.0.gpu‚48.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚159.696' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚39.924 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0pÊ?g¨‚(:Ü سêÔ ›ü€ disk.sda1.in‚0! disk.nvme0n1.out‚ 32324.734375 disk.sda2.in‚ 4.30078125 disk.rbd0.out‚ 1.68359375 disk.sdb1.in‚0# proc.memory.rssMB‚ 3002.45703125) proc.memory.percent‚18.32554340362549 disk.sdb2.in‚0.828125 disk.rbd2.out‚ 1.845703125 proc.cpu.threads‚17 disk.rbd0.in‚0 disk.sdb1.out‚0$ memory_percent‚92.06225872039795) proc.memory.availableMB‚ 1300.51953125 disk.dm-0.out‚ 75258.7109375 disk.sda.out‚ 740.845703125 cpu‚25.202463163181086 disk.sda1.out‚0 disk.sdb.in‚0.828125 disk.nvme1n1.in‚ 498.80078125" disk.nvme1n1.out‚ 42933.9765625 network.recv‚ 2.073765e+06# disk./.usageGB‚2712.95841217041 disk.md0p1.in‚ 5.12890625 disk.sdb.out‚ 740.845703125 disk.md0.in‚ 5.12890625 disk.md0.out‚ 713.8046875 network.sent‚ 1.395547e+07) disk./.usagePercent‚75.85545266559176 disk.sdb2.out‚ 740.845703125 disk.sda.in‚ 4.30078125 disk.sda2.out‚ 740.845703125 disk.rbd2.in‚0 disk.dm-0.in‚ 1102.24609375 disk.md0p1.out‚ 713.8046875 disk.nvme0n1.in‚ 603.44531257˜è­”©‚(:‰ ç³êÔüÛ»ÿ gpu.0.gpu‚42.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚136.743' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚34.18575 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0YB3sª‚(:è ç³êÔ¢°Øÿ disk.nvme1n1.in‚ 498.80078125 disk.dm-0.in‚ 1102.24609375 disk.sdb.in‚0.828125 network.sent‚ 1.3959137e+07! disk.sda2.out‚742.33251953125 disk.md0.in‚ 5.12890625 disk.md0.out‚ 715.23828125 disk.rbd2.in‚0 disk.sda2.in‚ 4.30078125 disk.dm-0.out‚ 75391.7109375 disk.nvme0n1.in‚ 603.4453125 disk.sdb.out‚742.33251953125# proc.memory.rssMB‚ 3002.45703125 network.recv‚ 2.077588e+06% disk./.usageGB‚2712.5283012390137 disk.sdb1.out‚0 disk.sda1.in‚0 disk.md0p1.out‚ 715.23828125# disk.nvme0n1.out‚32324.73828125 disk.sda.out‚742.33251953125' proc.memory.availableMB‚ 1300.765625 disk.rbd0.in‚0 disk.sdb1.in‚0 disk.rbd2.out‚ 1.845703125 disk.sda1.out‚0 disk.sda.in‚ 4.30078125$ memory_percent‚92.06075668334961 cpu‚25.20200557502187 proc.cpu.threads‚17 disk.sdb2.in‚0.828125( disk./.usagePercent‚75.8434265839345 disk.rbd0.out‚ 1.68359375# disk.nvme1n1.out‚43066.97265625! disk.sdb2.out‚742.33251953125 disk.md0p1.in‚ 5.12890625) proc.memory.percent‚18.32554340362549§ îx2«- _wandbruntime‚6478  _runtime‚6478çckŠÆ¬À & lr-AdamW/pg1‚0.0005569368808675887 & lr-AdamW/pg2‚0.0005569368808675887 trainer/global_step‚5599 $ _timestamp‚1.788516853309617e+09 _runtime‚6478.779954067 _step‚456S—ùŸË­Å & lr-AdamW/pg2‚0.0005569368808675887 trainer/global_step‚5599 $ _timestamp‚1.788516853309617e+09 _runtime‚6478.779954067 _step‚456 & lr-AdamW/pg1‚0.0005569368808675887È;’E§2®- _wandbruntime‚6478  _runtime‚6478gâS-¯ôO TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26090 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955591168 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955591168 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407872 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 255608629 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚520817975941632 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26088 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚465812078405632 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6356140567040 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚33523479 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 105214012 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1212 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚618 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 214179840 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 140293533 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 238119936 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 115289008 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6357039974912 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 398533632 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 _runtime‚ 6478.78486871 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 115289008 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 115289006 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6333338544806 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6332441133386 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177047040 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚33524097 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 140293533 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚71690535 GADeviceStatsMonitor.on_train_batch_staêZcCÜrt/oversize_segments.current‚0 _step‚457 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚459455038430720 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚465811160994816 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6356140567040 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 140319621 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 $ _timestamp‚1.788516853310644e+09 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450780 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214665 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917410816 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26088 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407872 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚520818214061568 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007241 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚465812078405632 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚449252149837968 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚514181226271744 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚449252166877328 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚465811160994816 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚71690533 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚514181202331648 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6636987789824 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0˜Ÿïwÿ'°ùO QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410816 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 105214632 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1187 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚620 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 105214012 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1212 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚618 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 214179840 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 140293533 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 238119936 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 115289008 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6357039974912 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 398533632 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 _runtime‚ 6478.78486871 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 115289008 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 115289006 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6333338544806 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6332441133386 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177047040 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚33524097 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 140293533 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚71690535 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _step‚457 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚459455038430720 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚465811160994816 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6356140567040 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 140319621 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 $ _timestamp‚1.788516853310644e+09 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450780 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214665 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917410816 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26088 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407872 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚520818214061568 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007241 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚465812078405632 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚449252149837968 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚514181226271744 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚449252166877328 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚465811160994816 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚71690533 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚514181202331648 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6636987789824 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26090 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955591168 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955591168 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407872 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 255608629 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚520817975941632 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26088 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚465812078405632 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6356140567040 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚33523479 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚465892821517824 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6638158608896 trainer/global_step‚5599 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177047040 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚459534650723328 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26114 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚622 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚514270521552384 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 105232088 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 255653144 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚465892821517824 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1187 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚520908704101376 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450988 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899421184 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 255627028 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚33529799 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚449330129932096 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 _runtime‚6479.758218935 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 115308906 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 224652288 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6637933956608 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 % _timestamp‚1.7885168542874286e+09 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6358152791552 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 _step‚458 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚455663679726800 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6334447206332 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899421184 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 140318124 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚449330146971456 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 248592384 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 398533632 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6357253370368 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 255653144 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 140344238 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚71702909 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955591168 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177047040 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 105232710 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚514270545492480 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚459534650723328 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728Õd/žÊ ³…N WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚465892821517824 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1187 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚520908704101376 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450988 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899421184 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 255627028 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚33529799 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚449330129932096 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 _runtime‚6479.758218935 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 115308906 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 224652288 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6637933956608 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 % _timestamp‚1.7885168542874286e+09 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6358152791552 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 _step‚458 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚455663679726800 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6334447206332 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899421184 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 140318124 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚449330146971456 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 248592384 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 398533632 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6357253370368 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 255653144 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 140344238 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚71702909 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955591168 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177047040 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 105232710 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚514270545492480 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚459534650723328 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 140344238 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚465891904093696 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚71702911 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 255627028 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚465892821517824 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6638158608896 trainer/global_step‚5599 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177047040 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 E?DeviceStatsMonito‰’ Ar.on_train_batch_end/oversize_segments.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚459534650723328 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26114 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚622 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚514270521552384 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 105232088 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 255653144Êïp¼I2´- _wandbruntime‚6479  _runtime‚6479ˆq¬°XµÒ /loss/FRt1/detection_loss‚2.0069174185579994 epoch‚4 1loss/SRqqt3/assignment_loss‚1.089676200884624 2loss/FBt1/assignment_loss‚0.030794083192391655 .loss/FRt4/detection_loss‚3.111525073647499 1loss/SRqqt3/detection_loss‚1.5191296811984933 0loss/FRt2/assignment_loss‚0.7281516829262609 0loss/SRqqt1/detection_loss‚1.595833380240947 trainer/global_step‚5599 /loss/FRt1/assignment_loss‚3.062578864892324 0loss/FRt3/assignment_loss‚2.2240334024616315 2loss/SRqqt1/assignment_loss‚1.3442249167710543 .loss/FRt2/detection_loss‚1.694834151831658 .loss/FBt1/detection_loss‚2.626468665070004 %loss/total_loss‚39.64932049001824 0loss/SRqqt2/detection_loss‚1.030907115102678 /loss/FBt2/detection_loss‚0.5634045020184096 .loss/FBt4/detection_loss‚2.249654605108149 $ _timestamp‚1.788516854290152e+09 .loss/FRt3/detection_loss‚2.207039765575353 1loss/SRqqt4/detection_loss‚2.4409940679158484 2loss/SRqqt4/assignment_loss‚1.4517942166754179 1loss/FBt2/assignment_loss‚0.11588306443451368 0loss/FBt3/assignment_loss‚0.7356287565531342 /loss/FBt3/detection_loss‚3.2779529094696045 _runtime‚6479.760615916 _step‚459 /loss/FRt4/assignment_loss‚3.326925794283549 2loss/SRqqt2/assignment_loss‚0.8854796957271849 1loss/FBt4/assignment_loss‚0.32948847547951277‰ârŸ]¶× /loss/FRt1/assignment_loss‚3.062578864892324 0loss/FRt3/assignment_loss‚2.2240334024616315 2loss/SRqqt1/assignment_loss‚1.3442249167710543 .loss/FRt2/detection_loss‚1.694834151831658 .loss/FBt1/detection_loss‚2.626468665070004 %loss/total_loss‚39.64932049001824 0loss/SRqqt2/detection_loss‚1.030907115102678 /loss/FBt2/detection_loss‚0.5634045020184096 .loss/FBt4/detection_loss‚2.249654605108149 $ _timestamp‚1.788516854290152e+09 .loss/FRt3/detection_loss‚2.207039765575353 1loss/SRqqt4/detection_loss‚2.4409940679158484 2loss/SRqqt4/assignment_loss‚1.4517942166754179 1loss/FBt2/assignment_loss‚0.11588306443451368 0loss/FBt3/assignment_loss‚0.7356287565531342 /loss/FBt3/detection_loss‚3.2779529094696045 _runtime‚6479.760615916 _step‚459 /loss/FRt4/assignment_loss‚3.326925794283549 2loss/SRqqt2/assignment_loss‚0.8854796957271849 1loss/FBt4/assignment_loss‚0.32948847547951277 /loss/FRt1/detection_loss‚2.0069174185579994 epoch‚4 1loss/SRqqt3/assignment_loss‚1.089676200884624 2loss/FBt1/assignment_loss‚0.030794083192391655 .loss/FRt4/detection_loss‚3.111525073647499 1loss/SRqqt3/detection_loss‚1.5191296811984933 0loss/FRt2/assignment_loss‚0.7281516829262609 0loss/SRqqt1/detection_loss‚1.595833380240947 trainer/global_step‚5599ˈ|+”·‚(:‰ ö³êÔêºþ gpu.0.gpu‚16.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚176.461' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚44.11525 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Úk«f¸‚(:Û ö³êÔí˜Íÿ) disk./.usagePercent‚75.84341154479947 disk.nvme0n1.in‚ 603.4453125 disk.sda2.out‚ 744.22265625 disk.sda1.in‚0 disk.rbd2.in‚0 disk.sdb2.out‚ 744.22265625 network.recv‚ 2.082522e+06 disk.nvme1n1.in‚ 498.80078125 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.md0.in‚ 5.12890625 disk.sda1.out‚0 disk.sdb2.in‚0.828125$ disk./.usageGB‚2712.527763366699 disk.md0p1.in‚ 5.12890625 disk.sda2.in‚ 4.30078125 disk.md0.out‚ 717.0859375 disk.sdb.out‚ 744.22265625 disk.sda.out‚ 744.22265625# disk.nvme1n1.out‚43178.18359375 proc.cpu.threads‚17# disk.nvme0n1.out‚32324.82421875 disk.rbd0.in‚0 disk.rbd2.out‚ 1.845703125 network.sent‚ 1.3978296e+07 disk.dm-0.in‚ 1102.24609375 disk.dm-0.out‚ 75503.0078125 disk.sdb.in‚0.828125 disk.sda.in‚ 4.30078125 disk.rbd0.out‚ 1.68359375 disk.md0p1.out‚ 717.0859375$ memory_percent‚92.06535816192627# proc.memory.rssMB‚ 3002.45703125) proc.memory.percent‚18.32554340362549 cpu‚25.2015020065733) proc.memory.availableMB‚ 1300.01171875"Ó‡ž¹‚(:“ …´êÔ–ñÏÿ gpu.0.gpu‚51.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚37.0 gpu.0.powerWatts‚126.497' gpu.0.enforcedPowerLimitWatts‚400.0) gpu.0.powerPercent‚31.624249999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÍW3;^º‚(:Ó …´êÔ ²¤€ disk.dm-0.in‚ 1102.24609375 disk.sda2.in‚ 4.30078125# proc.memory.rssMB‚ 3002.45703125 disk.rbd2.in‚0 disk.sda1.out‚0 disk.sdb.out‚ 746.09375" disk.nvme1n1.out‚ 43379.0078125 disk.sdb1.out‚0 proc.cpu.threads‚17) disk./.usagePercent‚75.84099419049947 disk.sdb2.in‚0.828125 disk.md0p1.in‚ 5.12890625 disk.md0p1.out‚ 718.88671875 disk.sdb.in‚0.828125 disk.rbd0.in‚0 disk.sdb1.in‚0 disk.md0.out‚ 718.88671875 network.sent‚ 1.4074667e+07 disk.nvme0n1.in‚ 603.4453125$ memory_percent‚92.06264019012451) proc.memory.percent‚18.32554340362549 cpu‚25.201071283121554 disk.dm-0.out‚75703.94140625$ disk./.usageGB‚2712.441307067871 disk.sda.out‚ 746.09375 disk.sda2.out‚ 746.09375 disk.nvme1n1.in‚ 498.80078125 disk.rbd2.out‚ 1.845703125 disk.sda1.in‚0 disk.rbd0.out‚ 1.68359375) proc.memory.availableMB‚ 1300.45703125 network.recv‚ 2.087501e+06 disk.sdb2.out‚ 746.09375# disk.nvme0n1.out‚32324.93359375 disk.sda.in‚ 4.30078125 disk.md0.in‚ 5.12890625gåk“»‚(:ˆ ”´êÔ¢’Ùý gpu.0.gpu‚55.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚39.0 gpu.0.powerWatts‚172.118' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.0295 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ê @l¼‚(:á ”´êÔÐ…âþ$ memory_percent‚92.06266403198242 disk.sdb2.out‚747.5048828125 disk.md0.in‚ 5.12890625 disk.md0p1.out‚ 720.2421875# disk.nvme0n1.out‚32396.01953125 disk.sda.out‚747.5048828125# proc.memory.rssMB‚ 3002.45703125) disk./.usagePercent‚75.83725040582199 disk.sda.in‚ 4.30078125 disk.sdb1.in‚0 network.recv‚ 2.091778e+06% disk./.usageGB‚2712.3074111938477 disk.md0p1.in‚ 5.12890625 disk.nvme0n1.in‚ 603.4453125 disk.sdb.out‚747.5048828125 disk.sdb1.out‚0 disk.md0.out‚ 720.2421875 disk.rbd2.out‚ 1.845703125 disk.dm-0.out‚75812.70703125) proc.memory.percent‚18.32554340362549 proc.cpu.threads‚17 disk.sdb2.in‚0.828125 disk.dm-0.in‚ 1102.24609375 disk.rbd0.in‚0 disk.nvme1n1.in‚ 498.80078125 disk.rbd2.in‚0 cpu‚25.200622091610402 network.sent‚ 1.4078501e+07 disk.sda1.in‚0 disk.sda1.out‚0 disk.sdb.in‚0.828125' proc.memory.availableMB‚ 1300.453125 disk.sda2.in‚ 4.30078125 disk.sda2.out‚747.5048828125 disk.rbd0.out‚ 1.68359375 disk.nvme1n1.out‚ 43416.6875RoC픽‚(:‰ £´êÔõÃïý gpu.0.gpu‚24.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚123.095' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚30.77375 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0/Ö ×g¾‚(:Ü £´êÔÑððþ disk.md0.in‚ 5.12890625 disk.dm-0.out‚ 75884.046875( proc.memory.availableMB‚ 1300.5859375) proc.memory.percent‚18.32554340362549 disk.sdb2.out‚749.0263671875 network.sent‚ 1.4084408e+07 disk.sdb1.out‚0 disk.md0.out‚ 721.703125 disk.sda1.in‚0$ memory_percent‚92.06185340881348 network.recv‚ 2.095923e+06 disk.sda2.out‚749.0263671875) disk./.usagePercent‚75.83616865470522% disk./.usageGB‚2712.2687225341797 disk.nvme1n1.in‚ 498.80078125 disk.nvme1n1.out‚ 43417.28125 disk.rbd2.in‚0 disk.nvme0n1.in‚ 603.4453125! disk.nvme0n1.out‚ 32466.765625 disk.sdb2.in‚0.828125 disk.sda.out‚749.0263671875 disk.rbd2.out‚ 1.845703125 disk.dm-0.in‚ 1102.24609375 disk.md0p1.in‚ 5.12890625 disk.sdb.in‚0.828125 disk.sda.in‚ 4.30078125 cpu‚25.20027619336495 proc.cpu.threads‚17 disk.sdb1.in‚0 disk.sda1.out‚0 disk.sda2.in‚ 4.30078125 disk.rbd0.out‚ 1.68359375 disk.sdb.out‚749.0263671875 disk.md0p1.out‚ 721.703125 disk.rbd0.in‚0# proc.memory.rssMB‚ 3002.45703125=çg2¿- _wandbruntime‚6532  _runtime‚6532¯ÕïIÆÀÀ _step‚460 & lr-AdamW/pg1‚0.0005510153818700209 & lr-AdamW/pg2‚0.0005510153818700209 trainer/global_step‚5649 $ _timestamp‚1.788516906926039e+09 _runtime‚6532.396480346±cDËÁÅ $ _timestamp‚1.788516906926039e+09 _runtime‚6532.396480346 _step‚460 & lr-AdamW/pg1‚0.0005510153818700209 & lr-AdamW/pg2‚0.0005510153818700209 trainer/global_step‚5649ÌN Œd2Â- _wandbruntime‚6532  _runtime‚6532¹=Õúü'ÃöO QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 141552523 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚518647187308544 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411420 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955591168 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚463436535205376 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177047040 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚525342489920000 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26088 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚469848330352640 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407872 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450780 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚453151171583728 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚469848330352640 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 257810341 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 116283906 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407872 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6388785201914 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6411795147264 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26088 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1187 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6695080043008 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 % _timestamp‚1.7885169069273145e+09 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410816 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚463436553208320 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 106119436 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚463436553208320 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚72309335 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007241 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚463436535205376 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 106120036 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚33810701 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6387887790494 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 141526435 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955591168 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚5649 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚600 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6412694555136 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚459539956785642 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚72309333 _runtime‚6532.397924094 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 116283908 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 141526435 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214665 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 116283906 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 257836431 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 106119436 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚463436553208320 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚72309335 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007241 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚463436535205376 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 106120036 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚33810701 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6387887790494 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 141526435 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955591168 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚5649 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚600 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6412694555136 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚459539956785642 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚72309333 _runtime‚6532.397924094 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 116283908 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 141526435 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214665 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 116283906 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 257836431 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚469848330352640 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407872 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450780 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚453151171583728 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚469848330352640 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 257810341 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 116283906 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407872 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6388785201914 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6411795147264 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26088 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1187 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6695080043008 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 % _timestamp‚1.7885169069273145e+09 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410816 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚463436553208320 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0ÍeÀ†l2Å- _wandbruntime‚6533  _runtime‚6533uv”Z8ÆÿM =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 141551026 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚518736482589184 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177047040 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 216263680 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚459619045541296 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚469929073451520 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 141577140 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 106137453 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚525432735951872 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6696253362688 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 _step‚462 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6412907950592 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚632 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚469929990875648 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450988 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 257854830 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 $ _timestamp‚1.788516907979395e+09 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 116303806 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 116303804 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 106138085 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6696469626368 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6389893863440 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 116303804 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26114 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 141551026 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6413807371776 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899421184 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899421184 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6413807371776 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚453229151677856 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚6533.450379721 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 257854830 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955591168 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚630 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411628 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚463516165500928 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚33816374 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚525432976155648 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6388996451812 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚72321709 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 257880946 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 141577140 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚518736506529280 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 trainer/global_step‚5649 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚463516165500928 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 398533632 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6412907950592 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1187 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚463516183503872 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚459618131090308 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917424128 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31686 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚453229134638496 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 240203776 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096$7 'Ç„N S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚469929073451520 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 141577140 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 106137453 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚525432735951872 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6696253362688 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 _step‚462 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6412907950592 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚632 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚469929990875648 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450988 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 257854830 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 $ _timestamp‚1.788516907979395e+09 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 116303806 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 116303804 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 106138085 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6696469626368 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6389893863440 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 116303804 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26114 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 141551026 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6413807371776 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899421184 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899421184 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6413807371776 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚453229151677856 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚6533.450379721 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 257854830 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955591168 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚630 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411628 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚463516165500928 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚33816374 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚525432976155648 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6388996451812 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚72321709 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 257880946 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 141577140 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚518736506529280 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 trainer/global_step‚5649 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚463516165500928 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 398533632 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6412907950592 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1187 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚463516183503872 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚459618131090308 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917424128 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31686 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚453229134638496 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 240203776 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 141551026 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚518736482589184 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177047040 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 216263680 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚459619045541296Îç4Ê2È- _wandbruntime‚6533  _runtime‚65331_Û0UÉÏ 1loss/FBt1/assignment_loss‚0.18776332916136443 1loss/FBt4/assignment_loss‚0.17692532984565332 -loss/FRt3/detection_loss‚2.13434820358331 .loss/FBt4/detection_loss‚2.322387170791626 epoch‚4 /loss/FRt1/assignment_loss‚2.212800583418678 1loss/SRqqt2/assignment_loss‚1.078063934886207 2loss/SRqqt3/assignment_loss‚1.1520831247430239 .loss/FRt2/detection_loss‚1.488793490759351 1loss/SRqqt1/detection_loss‚1.5643825916780365 1loss/SRqqt3/detection_loss‚1.5225862028883463 1loss/SRqqt4/detection_loss‚2.3139930376500795 .loss/FRt4/assignment_loss‚3.56528312486151 2loss/SRqqt4/assignment_loss‚1.6124688643171932 _step‚463 2loss/SRqqt1/assignment_loss‚1.0217376114800572 1loss/FBt2/assignment_loss‚0.10262773048532221 /loss/FBt2/detection_loss‚0.6795409358250843 &loss/total_loss‚39.038739083009744 /loss/FBt1/detection_loss‚2.5608957282134464 trainer/global_step‚5649 1loss/SRqqt2/detection_loss‚0.9267705890039603 _runtime‚6533.452044759 /loss/FRt2/assignment_loss‚1.498313884301619 0loss/FRt3/assignment_loss‚2.5026944540441036 0loss/FBt3/assignment_loss‚0.1083732458112839 .loss/FRt1/detection_loss‚1.666616352143672 .loss/FRt4/detection_loss‚3.261506920275481 .loss/FBt3/detection_loss‚3.377782642841339 # _timestamp‚1.78851690798181e+09Y±]mZÊÔ 1loss/FBt1/assignment_loss‚0.18776332916136443 1loss/FBt4/assignment_loss‚0.17692532984565332 -loss/FRt3/detection_loss‚2.13434820358331 .loss/FBt4/detection_loss‚2.322387170791626 epoch‚4 /loss/FRt1/assignment_loss‚2.212800583418678 1loss/SRqqt2/assignment_loss‚1.078063934886207 2loss/SRqqt3/assignment_loss‚1.1520831247430239 .loss/FRt2/detection_loss‚1.488793490759351 1loss/SRqqt1/detection_loss‚1.5643825916780365 1loss/SRqqt3/detection_loss‚1.5225862028883463 1loss/SRqqt4/detection_loss‚2.3139930376500795 .loss/FRt4/assignment_loss‚3.56528312486151 2loss/SRqqt4/assignment_loss‚1.6124688643171932 _step‚463 2loss/SRqqt1/assignment_loss‚1.0217376114800572 1loss/FBt2/assignment_loss‚0.10262773048532221 /loss/FBt2/detection_loss‚0.6795409358250843 &loss/total_loss‚39.038739083009744 /loss/FBt1/detection_loss‚2.5608957282134464 trainer/global_step‚5649 1loss/SRqqt2/detection_loss‚0.9267705890039603 _runtime‚6533.452044759 /loss/FRt2/assignment_loss‚1.498313884301619 0loss/FRt3/assignment_loss‚2.5026944540441036 0loss/FBt3/assignment_loss‚0.1083732458112839 .loss/FRt1/detection_loss‚1.666616352143672 .loss/FRt4/detection_loss‚3.261506920275481 .loss/FBt3/detection_loss‚3.377782642841339 # _timestamp‚1.78851690798181e+09ÏHsw”Ë‚(:‰ ²´êÔ꯫þ gpu.0.gpu‚19.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚161.587' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚40.39675 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0qÜŽøbÌ‚(:× ²´êÔëÃÖþ disk.sda1.in‚0 disk.sdb1.in‚0 disk.md0.out‚ 723.68359375 disk.dm-0.in‚ 1102.24609375 disk.sda1.out‚0# proc.memory.rssMB‚ 3002.45703125 disk.md0p1.out‚ 723.68359375 disk.sdb.in‚0.828125 disk.md0.in‚ 5.12890625 disk.md0p1.in‚ 5.12890625" disk.nvme1n1.out‚ 43417.6953125 disk.sdb1.out‚0 disk.rbd2.in‚0 disk.sdb2.in‚0.828125) proc.memory.percent‚18.32554340362549 cpu‚25.19985838105368 disk.sda2.out‚ 751.06640625 network.recv‚ 2.100853e+06$ disk./.usageGB‚2712.458770751953! disk.nvme0n1.out‚ 32581.671875 disk.sdb.out‚ 751.06640625 disk.nvme1n1.in‚ 498.80078125% proc.memory.availableMB‚ 1300.5625 proc.cpu.threads‚17 disk.nvme0n1.in‚ 603.4453125 disk.sda.out‚ 751.06640625 disk.rbd0.out‚ 1.68359375 disk.rbd2.out‚ 1.845703125 disk.dm-0.out‚ 75999.3671875$ memory_percent‚92.06199645996094 disk.sda2.in‚ 4.30078125 network.sent‚ 1.4090824e+07 disk.sda.in‚ 4.30078125 disk.rbd0.in‚0) disk./.usagePercent‚75.84148248241553 disk.sdb2.out‚ 751.06640625ã,'F“Í‚(:ˆ Á´êÔ“¶¦ÿ gpu.0.gpu‚54.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚172.118' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.0295 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0R ¦Ür΂(:ç Á´êÔ‡ Öÿ disk.md0.out‚ 725.18359375 disk.md0p1.out‚ 725.18359375% disk./.usageGB‚2712.7087173461914 disk.sdb1.in‚0 disk.rbd2.out‚ 1.845703125 disk.sda1.in‚0 disk.dm-0.out‚76145.69921875 network.sent‚ 1.4197783e+07 disk.sdb.out‚752.61572265625! disk.sda2.out‚752.61572265625 disk.rbd0.out‚ 1.68359375 proc.cpu.threads‚17 disk.sdb.in‚0.828125 disk.rbd0.in‚0 disk.sdb1.out‚0 disk.rbd2.in‚0 disk.sdb2.in‚0.828125# disk.nvme0n1.out‚32727.88671875 disk.nvme1n1.out‚ 43417.8125 disk.md0.in‚ 5.12890625 disk.nvme0n1.in‚ 603.4453125( proc.memory.availableMB‚ 1300.4140625 cpu‚25.199470769094432 network.recv‚ 2.105396e+06! disk.sdb2.out‚752.61572265625 disk.sda1.out‚0 disk.sda2.in‚ 4.30078125# proc.memory.rssMB‚ 3002.45703125) proc.memory.percent‚18.32554340362549 disk.sda.in‚ 4.30078125 disk.sda.out‚752.61572265625 disk.nvme1n1.in‚ 498.80078125 disk.dm-0.in‚ 1102.24609375 disk.md0p1.in‚ 5.12890625$ memory_percent‚92.06290245056152( disk./.usagePercent‚75.8484710938012Éåüö”Ï‚(:‰ дêÔÑ©äý gpu.0.gpu‚21.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚37.0 gpu.0.powerWatts‚143.595' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚35.89875 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0'$‘…oЂ(:ä дêÔâ°Æÿ disk.sdb2.in‚0.828125# disk.nvme0n1.out‚32961.16015625 disk.sda.in‚ 4.30078125 disk.nvme1n1.in‚ 498.80078125 disk.md0.out‚ 726.69140625 disk.dm-0.in‚ 1102.24609375$ memory_percent‚92.06204414367676( proc.memory.availableMB‚ 1300.5546875 disk.sda.out‚754.1748046875 disk.rbd2.in‚0# proc.memory.rssMB‚ 3002.45703125 disk.sdb1.out‚0 disk.rbd2.out‚ 1.845703125) proc.memory.percent‚18.32554340362549 disk.sda2.out‚754.1748046875 disk.rbd0.in‚0 cpu‚25.199131881392578 network.sent‚ 1.420382e+07 disk.sdb2.out‚754.1748046875 disk.md0p1.in‚ 5.12890625 network.recv‚ 2.109873e+06 disk.rbd0.out‚ 1.68359375 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-0.out‚76379.37890625 disk.sdb.out‚754.1748046875% disk./.usageGB‚2713.0843925476074 disk.sda2.in‚ 4.30078125 disk.nvme1n1.out‚ 43418.21875 disk.md0p1.out‚ 726.69140625 disk.nvme0n1.in‚ 603.4453125 disk.sdb.in‚0.828125 proc.cpu.threads‚17) disk./.usagePercent‚75.85897512966507 disk.sdb1.in‚0 disk.md0.in‚ 5.12890625Éfúu“Ñ‚(:ˆ ß´êÔÝ©õý gpu.0.gpu‚34.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚178.914' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚44.7285 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Qû]…sÒ‚(:è ß´êÔÛÕ÷þ disk.sdb.in‚0.828125 proc.cpu.threads‚17 network.sent‚ 1.4208776e+07 disk.sda.out‚756.17236328125 disk.sda1.out‚0# proc.memory.rssMB‚ 3002.45703125 disk.md0p1.in‚ 5.12890625 network.recv‚ 2.113501e+06) disk./.usagePercent‚75.85894825121098% disk./.usageGB‚2713.0834312438965 disk.rbd0.out‚ 1.68359375 disk.nvme1n1.out‚ 43418.34375 disk.sdb1.out‚0! disk.sda2.out‚756.17236328125 disk.md0.in‚ 5.12890625 disk.rbd2.out‚ 1.845703125# disk.nvme0n1.out‚33172.04296875 disk.sdb.out‚756.17236328125 disk.sdb1.in‚0$ memory_percent‚92.06273555755615) proc.memory.availableMB‚ 1300.44140625 cpu‚25.198722685144503 disk.rbd0.in‚0 disk.md0.out‚ 728.6328125! disk.sdb2.out‚756.17236328125 disk.sda1.in‚0 disk.dm-0.in‚ 1102.24609375 disk.dm-0.out‚76590.38671875 disk.nvme0n1.in‚ 603.4453125 disk.sda2.in‚ 4.30078125) proc.memory.percent‚18.32554340362549 disk.sda.in‚ 4.30078125 disk.nvme1n1.in‚ 498.80078125 disk.rbd2.in‚0 disk.sdb2.in‚0.828125 disk.md0p1.out‚ 728.6328125>„ó‡2Ó- _wandbruntime‚6586  _runtime‚6586ó g‹ÄÔ¾ % lr-AdamW/pg1‚0.000545068423852539 % lr-AdamW/pg2‚0.000545068423852539 trainer/global_step‚5699 $ _timestamp‚1.788516960738681e+09 _runtime‚6586.209066595 _step‚464N1Œ¡ÉÕà _runtime‚6586.209066595 _step‚464 % lr-AdamW/pg1‚0.000545068423852539 % lr-AdamW/pg2‚0.000545068423852539 trainer/global_step‚5699 $ _timestamp‚1.788516960738681e+09ТXX2Ö- _wandbruntime‚6586  _runtime‚6586´oÕ×öO YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚467418067985920 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚473885499710464 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚529866171617792 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 107024947 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6443334447602 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7885169607398036e+09 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26088 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚523113148345344 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚529866420223488 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚467418049982976 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177047040 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26088 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚467418067985920 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411420 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 117278808 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6468349135360 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚619 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955591168 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚523113124405248 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407872 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 398533632 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26090 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.û*. 'large_pool.freed‚467418049982976 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 117278808 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚463493493698370 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6444231859022 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 224665600 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410816 trainer/global_step‚5699 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚473885499710464 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚34096814 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 260038143 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6467449727488 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 142759337 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 107025568 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚72928135 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚72928133 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6753271878144 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 142785425 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚473886417121280 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1212 _runtime‚6586.210853293 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450780 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407872 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6468349135360 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚619 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955591168 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚523113124405248 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407872 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 398533632 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26090 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚467418049982976 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 117278808 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚463493493698370 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6444231859022 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 224665600 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410816 trainer/global_step‚5699 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚473885499710464 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚34096814 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 260038143 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6467449727488 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 142759337 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 107025568 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚72928135 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚72928133 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6753271878144 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 142785425 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚473886417121280 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1212 _runtime‚6586.210853293 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450780 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407872 JDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 260108748 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6468562530816 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 117298706 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31686 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917424128 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177047040 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚473967160233472 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 260082632 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 142810042 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 398533632 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6445340520548 _runtime‚6587.202073799 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚467497680278528 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899421184 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚34103158 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚523202467566080 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6469461952000 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6468562530816 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 107043046 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 142783928 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 214166528 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 260108748 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚473966242809344 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚623 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚463572582453816 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 117298704 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6444443108920 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚467497680278528 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚467497698281472 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6469461952000 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚72940511 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚72940509 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚621 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171t7âÐ~Û„N FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 117298704 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6444443108920 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚467497680278528 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚467497698281472 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6469461952000 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚72940511 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚72940509 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚621 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 117298704 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26114 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚457128156384256 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007241 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 trainer/global_step‚5699 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 142783928 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 107043669 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899421184 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6754231143936 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450988 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411628 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚467497698281472 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177047040 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6754445310464 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 117298706 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 260108748 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6468562530816 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 117298706 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31686 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917424128 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177047040 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 T@DeviceStatsMonitor.on_train_batch_end/aGÉýŒ ctive_bytes.all.allocated‚473967160233472 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 260082632 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 142810042 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 398533632 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6445340520548 _runtime‚6587.202073799 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚467497680278528 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899421184 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚34103158 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚523202467566080 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6469461952000 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6468562530816 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 107043046 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 142783928 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 214166528 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 260108748 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚473966242809344 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚623 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚463572582453816 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984Ò #.2Ü- _wandbruntime‚6587  _runtime‚65875½¿aÝÛ 0loss/FRt3/assignment_loss‚2.1314954301144216 0loss/FRt4/assignment_loss‚3.5263392251470815 /loss/FBt4/detection_loss‚2.5215092062950135 &loss/total_loss‚38.443778146663846 trainer/global_step‚5699 0loss/FBt3/assignment_loss‚0.5666895106181843 /loss/FRt2/detection_loss‚1.8891287642634578 2loss/SRqqt3/assignment_loss‚1.5393075038541535 1loss/FBt4/assignment_loss‚0.31224649871546717 /loss/FRt4/detection_loss‚3.1302812669588174 0loss/FRt2/assignment_loss‚0.5675999919573466 % _timestamp‚1.7885169617330122e+09 _step‚467 0loss/FRt1/assignment_loss‚2.9813832973324974 1loss/FBt2/assignment_loss‚0.07994343379428216 1loss/SRqqt2/detection_loss‚0.9268208134013254 1loss/SRqqt3/detection_loss‚1.4106235820494715 2loss/SRqqt1/assignment_loss‚1.3572823359941444 2loss/SRqqt4/assignment_loss‚1.4698466621339321 .loss/FRt3/detection_loss‚2.071305770862927 1loss/SRqqt4/detection_loss‚2.7972890680486504 /loss/FBt1/detection_loss‚1.9078482925891875 /loss/FRt1/detection_loss‚1.9074664452374597 2loss/SRqqt2/assignment_loss‚0.9509754012955504 1loss/FBt1/assignment_loss‚0.13990481712826294 0loss/SRqqt1/detection_loss‚1.711069050927957 /loss/FBt2/detection_loss‚0.5860733726931115 /loss/FBt3/detection_loss‚1.9613484052511363 epoch‚4 _runtime‚6587.203844107¿œ¥fÞà .loss/FRt3/detection_loss‚2.071305770862927 1loss/SRqqt4/detection_loss‚2.7972890680486504 /loss/FBt1/detection_loss‚1.9078482925891875 /loss/FRt1/detection_loss‚1.9074664452374597 2loss/SRqqt2/assignment_loss‚0.9509754012955504 1loss/FBt1/assignment_loss‚0.13990481712826294 0loss/SRqqt1/detection_loss‚1.711069050927957 /loss/FBt2/detection_loss‚0.5860733726931115 /loss/FBt3/detection_loss‚1.9613484052511363 epoch‚4 _runtime‚6587.203844107 0loss/FRt3/assignment_loss‚2.1314954301144216 0loss/FRt4/assignment_loss‚3.5263392251470815 /loss/FBt4/detection_loss‚2.5215092062950135 &loss/total_loss‚38.443778146663846 trainer/global_step‚5699 0loss/FBt3/assignment_loss‚0.5666895106181843 /loss/FRt2/detection_loss‚1.8891287642634578 2loss/SRqqt3/assignment_loss‚1.5393075038541535 1loss/FBt4/assignment_loss‚0.31224649871546717 /loss/FRt4/detection_loss‚3.1302812669588174 0loss/FRt2/assignment_loss‚0.5675999919573466 % _timestamp‚1.7885169617330122e+09 _step‚467 0loss/FRt1/assignment_loss‚2.9813832973324974 1loss/FBt2/assignment_loss‚0.07994343379428216 1loss/SRqqt2/detection_loss‚0.9268208134013254 1loss/SRqqt3/detection_loss‚1.4106235820494715 2loss/SRqqt1/assignment_loss‚1.3572823359941444 2loss/SRqqt4/assignment_loss‚1.4698466621339321Óìq±ï“ß‚(:ˆ î´êÔ¡êƒ gpu.0.gpu‚47.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚160.922' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚40.2305 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ð*`Òià‚(:Þ î´êÔöî‹‚ network.recv‚ 2.118249e+06 disk.rbd0.out‚ 1.68359375 disk.nvme1n1.in‚ 498.80078125 disk.sdb1.out‚0) proc.memory.availableMB‚ 1300.38671875 disk.sda2.out‚ 757.80078125# disk.nvme0n1.out‚33410.34765625 disk.sdb.in‚0.828125 disk.md0p1.out‚ 730.20703125 disk.sda.in‚ 4.30078125 disk.rbd0.in‚0# memory_percent‚92.0630693435669# proc.memory.rssMB‚ 3002.45703125) disk./.usagePercent‚75.85894686462407 disk.sda2.in‚ 4.30078125 disk.md0.in‚ 5.12890625) proc.memory.percent‚18.32554340362549 proc.cpu.threads‚17 disk.rbd2.out‚ 1.845703125 disk.sdb2.in‚0.828125 disk.sdb2.out‚ 757.80078125$ disk./.usageGB‚2713.083381652832" disk.nvme1n1.out‚ 43418.6015625 disk.sdb1.in‚0 disk.sdb.out‚ 757.80078125 disk.sda.out‚ 757.80078125 disk.md0.out‚ 730.20703125 disk.rbd2.in‚0 disk.sda1.in‚0 disk.dm-0.out‚76828.94921875 disk.md0p1.in‚ 5.12890625 cpu‚25.198323162590217 network.sent‚ 1.4215098e+07 disk.sda1.out‚0 disk.dm-0.in‚ 1102.24609375 disk.nvme0n1.in‚ 603.4453125׃çö“á‚(:ˆ ý´êÔŸ¢Ûÿ gpu.0.gpu‚53.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚169.942' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.4855 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0B®Ð4mâ‚(:â ý´êÔ¥€ disk.sda1.in‚0 disk.nvme0n1.in‚ 603.4453125 disk.sda.out‚759.1630859375 disk.rbd0.out‚ 1.68359375 disk.md0.out‚ 731.50390625 disk.dm-0.in‚ 1102.24609375 disk.md0p1.in‚ 5.12890625 disk.nvme1n1.out‚ 43418.71875 disk.sda1.out‚0 disk.sda.in‚ 4.30078125 disk.sda2.out‚759.1630859375 cpu‚25.197928363859322 disk.sdb1.out‚0 disk.sdb2.out‚759.1630859375 disk.rbd0.in‚0 network.recv‚ 2.122646e+06$ disk./.usageGB‚2712.811367034912 disk.md0.in‚ 5.12890625 disk.rbd2.out‚ 1.845703125 disk.sda2.in‚ 4.30078125 disk.sdb.out‚759.1630859375 network.sent‚ 1.4324287e+07 disk.nvme1n1.in‚ 498.80078125 disk.sdb1.in‚0 disk.rbd2.in‚0# disk.nvme0n1.out‚33677.52734375# proc.memory.rssMB‚ 3002.45703125 proc.cpu.threads‚17 disk.sdb2.in‚0.828125 disk.dm-0.out‚77096.24609375 disk.sdb.in‚0.828125& proc.memory.availableMB‚ 1300.59375) proc.memory.percent‚18.32554340362549) disk./.usagePercent‚75.85134122206007 disk.md0p1.out‚ 731.50390625$ memory_percent‚92.06180572509766À¯ý"”ã‚(:‰ ŒµêÔ„Ïšÿ gpu.0.gpu‚54.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚168.715' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.17875 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0alüýpä‚(:å ŒµêÔʽêÿ network.recv‚ 2.127409e+06" disk.nvme1n1.out‚ 43419.1171875# proc.memory.rssMB‚ 3002.45703125$ disk./.usageGB‚2712.772647857666 disk.rbd2.in‚0 disk.md0p1.in‚ 5.12890625" disk.nvme0n1.out‚ 33715.6171875 disk.sdb1.in‚0) proc.memory.percent‚18.32554340362549) disk./.usagePercent‚75.85025861765902 disk.md0p1.out‚ 733.15234375 disk.sdb.in‚0.828125 disk.rbd0.in‚0 disk.rbd0.out‚ 1.68359375 cpu‚25.197597287723728 disk.dm-0.out‚ 77134.734375 disk.md0.out‚ 733.15234375 disk.sdb2.in‚0.828125 disk.sda1.in‚0 disk.nvme0n1.in‚ 603.4453125 disk.sdb.out‚760.87744140625 disk.sda2.in‚ 4.30078125 network.sent‚ 1.4330431e+07 disk.sda.out‚760.87744140625 disk.nvme1n1.in‚ 498.80078125 disk.dm-0.in‚ 1102.24609375 disk.md0.in‚ 5.12890625 disk.rbd2.out‚ 1.845703125! disk.sdb2.out‚760.87744140625$ memory_percent‚92.06180572509766 disk.sda.in‚ 4.30078125! disk.sda2.out‚760.87744140625 proc.cpu.threads‚17 disk.sda1.out‚0 disk.sdb1.out‚0& proc.memory.availableMB‚ 1300.59375(a°­2å- _wandbruntime‚6640  _runtime‚6640ð!CáÆæÀ _runtime‚ 6640.32774619 _step‚468 & lr-AdamW/pg1‚0.0005390972461522803 & lr-AdamW/pg2‚0.0005390972461522803 trainer/global_step‚5749 % _timestamp‚1.7885170148572907e+09Jlù€ËçÅ trainer/global_step‚5749 % _timestamp‚1.7885170148572907e+09 _runtime‚ 6640.32774619 _step‚468 & lr-AdamW/pg1‚0.0005390972461522803 & lr-AdamW/pg2‚0.0005390972461522803Ôª•üÓ2è- _wandbruntime‚6640  _runtime‚6640¨>eü'éöO G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚527579109382144 _step‚469 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410816 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚603 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1212 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007241 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 118273708 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚471399564760576 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 trainer/global_step‚5749 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6499678516130 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚460949180996528 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚477922669068288 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 262292035 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 118273708 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 107930778 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚34383843 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6498781104710 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26088 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6811665591296 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0(÷â‚(êûO _runtime‚6640.329024045 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚467448859512658 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚477922669068288 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚527579109382144 _step‚469 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410816 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚603 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1212 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007241 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 118273708 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚471399564760576 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 trainer/global_step‚5749 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6499678516130 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚460949180996528 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚477922669068288 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 262292035 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 118273708 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 107930778 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚34383843 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6498781104710 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26088 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6811665591296 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 JDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚73559311 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚467527033817324 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 144016830 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917424128 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚467527948268312 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007241 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.fvš"ö-$reed‚ 262310434 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 107948322 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6524217111040 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚534481274962432 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6500787177656 _runtime‚6641.363412394 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 240203776 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450988 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚527668428602880 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26114 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 262336550 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚471479213059072 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 262336550 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 118293606 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 118293606 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚461027144051296 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 144016830 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1187 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 107948911 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6499889766028 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 trainer/global_step‚5749 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 262310434 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚34389600 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26114 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚589 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6812846359552 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚478004329591296 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6525116532224 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6524217111040 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚534481034758656 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 % _timestamp‚1.7885170158924582e+09 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚471479195056128 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899421184 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 144042944 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚471479213059072 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177047040 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 _step‚470 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 216263680 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 118293604 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177047040 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 398533632 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6525116532224 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚478003412167168 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6812630095872+%- 'í…N RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26114 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 262336550 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚471479213059072 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 262336550 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 118293606 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 118293606 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚461027144051296 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 144016830 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1187 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 107948911 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6499889766028 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 trainer/global_step‚5749 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 262310434 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚34389600 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26114 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚589 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6812846359552 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚478004329591296 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6525116532224 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6524217111040 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚534481034758656 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 % _timestamp‚1.7885170158924582e+09 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚471479195056128 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899421184 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 144042944 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚471479213059072 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177047040 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 _step‚470 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 216263680 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 118293604 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177047040 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 398533632 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6525116532224 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚478003412167168 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6812630095872 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚73559311 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚467527033817324 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 144016830 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917424128 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚467527948268312 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007241 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 262310434 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 107948322 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6524217111040 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚534481274962432 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6500787177656 _runtime‚6641.363412394 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 240203776 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450988 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚527668428602880 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0Ö’p®T2î- _wandbruntime‚6641  _runtime‚6641×[[ïÕ /loss/FRt1/assignment_loss‚2.924674949475697 1loss/SRqqt1/assignment_loss‚1.521106130182743 /loss/FRt1/detection_loss‚2.2142844325676987 /loss/FBt4/detection_loss‚2.9249153648103987 _runtime‚6641.364499438 0loss/FRt2/assignment_loss‚1.7512898325920105 1loss/SRqqt2/assignment_loss‚0.954245522507091 0loss/FRt4/assignment_loss‚3.0622967858063546 2loss/SRqqt4/assignment_loss‚2.3882653266191483 /loss/FRt3/detection_loss‚2.1083032394448917 0loss/SRqqt4/detection_loss‚2.727613502078586 /loss/FBt2/detection_loss‚0.5128016975018885 0loss/FBt4/assignment_loss‚0.4581502979385732 /loss/FRt4/detection_loss‚3.4872778120793795 1loss/SRqqt1/detection_loss‚1.8600672724843026 %loss/total_loss‚41.52839513540962 epoch‚4 % _timestamp‚1.7885170158943932e+09 _step‚471 0loss/FBt1/assignment_loss‚0.1601250002568122 0loss/SRqqt2/detection_loss‚1.137948264715387 .loss/FBt1/detection_loss‚2.172418624162674 /loss/FBt3/detection_loss‚2.6040998697280884 0loss/FRt3/assignment_loss‚2.2790860017140706 2loss/SRqqt3/assignment_loss‚1.1964060235463205 1loss/FBt2/assignment_loss‚0.08097139871334177 1loss/FBt3/assignment_loss‚0.09821004027617164 /loss/FRt2/detection_loss‚1.5988865055482495 1loss/SRqqt3/detection_loss‚1.3049512406597372 trainer/global_step‚5749a|Tµ`ðÚ /loss/FRt1/assignment_loss‚2.924674949475697 1loss/SRqqt1/assignment_loss‚1.521106130182743 /loss/FRt1/detection_loss‚2.2142844325676987 /loss/FBt4/detection_loss‚2.9249153648103987 _runtime‚6641.364499438 0loss/FRt2/assignment_loss‚1.7512898325920105 1loss/SRqqt2/assignment_loss‚0.954245522507091 0loss/FRt4/assignment_loss‚3.0622967858063546 2loss/SRqqt4/assignment_loss‚2.3882653266191483 /loss/FRt3/detection_loss‚2.1083032394448917 0loss/SRqqt4/detection_loss‚2.727613502078586 /loss/FBt2/detection_loss‚0.5128016975018885 0loss/FBt4/assignment_loss‚0.4581502979385732 /loss/FRt4/detection_loss‚3.4872778120793795 1loss/SRqqt1/detection_loss‚1.8600672724843026 %loss/total_loss‚41.52839513540962 epoch‚4 % _timestamp‚1.7885170158943932e+09 _step‚471 0loss/FBt1/assignment_loss‚0.1601250002568122 0loss/SRqqt2/detection_loss‚1.137948264715387 .loss/FBt1/detection_loss‚2.172418624162674 /loss/FBt3/detection_loss‚2.6040998697280884 0loss/FRt3/assignment_loss‚2.2790860017140706 2loss/SRqqt3/assignment_loss‚1.1964060235463205 1loss/FBt2/assignment_loss‚0.08097139871334177 1loss/FBt3/assignment_loss‚0.09821004027617164 /loss/FRt2/detection_loss‚1.5988865055482495 1loss/SRqqt3/detection_loss‚1.3049512406597372 trainer/global_step‚5749×ãæcØ”ñ‚(:‰ ›µêÔ¾•šÿ gpu.0.gpu‚13.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚174.903' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.72575 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0űçóhò‚(:Ý ›µêÔÛŠž€ disk.md0p1.in‚ 5.12890625 disk.nvme1n1.in‚ 498.80078125 disk.sdb1.out‚0 disk.md0.out‚ 735.19921875 disk.sda1.in‚0 disk.sda2.out‚ 762.98046875( proc.memory.availableMB‚ 1300.3203125 disk.sda1.out‚0# disk.nvme1n1.out‚43419.21484375 disk.sdb1.in‚0 disk.md0.in‚ 5.12890625 disk.rbd2.out‚ 1.845703125 disk.sda2.in‚ 4.30078125 disk.rbd2.in‚0 cpu‚25.19722478313768) disk./.usagePercent‚75.85193948098473$ disk./.usageGB‚2712.832763671875 disk.sdb2.in‚0.828125 disk.dm-0.out‚ 77497.1015625# disk.nvme0n1.out‚34077.88671875 disk.sdb.in‚0.828125 disk.rbd0.out‚ 1.68359375 proc.cpu.threads‚17 disk.md0p1.out‚ 735.19921875 disk.nvme0n1.in‚ 603.4453125 disk.sda.out‚ 762.98046875 disk.rbd0.in‚0# proc.memory.rssMB‚ 3002.45703125 network.recv‚ 2.131605e+06 disk.sdb2.out‚ 762.98046875 disk.dm-0.in‚ 1102.24609375$ memory_percent‚92.06347465515137) proc.memory.percent‚18.32554340362549 network.sent‚ 1.4334074e+07 disk.sdb.out‚ 762.98046875 disk.sda.in‚ 4.30078125õ5-Ø‘ó‚(:† ªµêÔ½“˜ÿ gpu.0.gpu‚11.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚126.22' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚31.555 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0mÄÕÕgô‚(:Ü ªµêÔ¢µÙÿ) disk./.usagePercent‚75.85894345148704 disk.sda.out‚ 764.701171875 disk.md0p1.in‚ 5.12890625 disk.md0p1.out‚ 736.859375 disk.sdb.in‚0.828125 disk.sda2.out‚ 764.701171875 disk.rbd2.out‚ 1.845703125 disk.sdb2.out‚ 764.701171875 disk.sda1.in‚0 disk.sdb1.in‚0 disk.sda1.out‚0 network.sent‚ 1.4442267e+07 disk.sda.in‚ 4.30078125 disk.sdb1.out‚0 disk.md0.in‚ 5.12890625$ memory_percent‚92.06514358520508 proc.cpu.threads‚17 disk.sda2.in‚ 4.30078125 disk.rbd0.in‚0 disk.rbd2.in‚0# proc.memory.rssMB‚ 3002.45703125 cpu‚25.19686123974991 network.recv‚ 2.137456e+06% disk./.usageGB‚2713.0832595825195 disk.nvme1n1.in‚ 498.80078125 disk.dm-0.in‚ 1102.24609375 disk.dm-0.out‚ 77701.953125 disk.nvme0n1.in‚ 603.4453125# disk.nvme0n1.out‚34282.24609375 disk.sdb.out‚ 764.701171875 disk.rbd0.out‚ 1.68359375 disk.md0.out‚ 736.859375' proc.memory.availableMB‚ 1300.046875) proc.memory.percent‚18.32554340362549# disk.nvme1n1.out‚43419.70703125 disk.sdb2.in‚0.828125E ª¼jõ‚(:ß ¹µêÔþÉåþ$ memory_percent‚92.06511974334717 cpu‚25.19646680005154 disk.rbd0.in‚0) disk./.usagePercent‚75.84340258531478 disk.nvme0n1.in‚ 603.4453125# disk.nvme0n1.out‚34341.49609375 disk.sdb.out‚ 766.330078125 disk.sda2.out‚ 766.330078125 disk.rbd0.out‚ 1.68359375 disk.rbd2.in‚0 disk.rbd2.out‚ 1.845703125 disk.md0p1.in‚ 5.12890625$ disk./.usageGB‚2712.527442932129 disk.sdb1.out‚0 disk.md0p1.out‚ 738.41796875 network.sent‚ 1.4447561e+07 disk.sda2.in‚ 4.30078125# disk.nvme1n1.out‚43419.81640625 disk.sda1.in‚0) proc.memory.availableMB‚ 1300.05078125 proc.cpu.threads‚17 disk.sda.out‚ 766.330078125 disk.sdb.in‚0.828125 disk.md0.out‚ 738.41796875 disk.sda1.out‚0 network.recv‚ 2.140896e+06 disk.sda.in‚ 4.30078125 disk.sdb1.in‚0 disk.sdb2.in‚0.828125# proc.memory.rssMB‚ 3002.45703125) proc.memory.percent‚18.32554340362549 disk.nvme1n1.in‚ 498.80078125 disk.md0.in‚ 5.12890625 disk.sdb2.out‚ 766.330078125 disk.dm-0.in‚ 1102.24609375 disk.dm-0.out‚ 77761.3125³˜%’ö‚(:‡ ¹µêÔ‚‘ ÿ gpu.0.gpu‚55.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚173.344' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.336 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0A:G’÷‚(:‡ ȵêÔ£þóý gpu.0.gpu‚52.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚162.204' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚40.551 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0b=k=hø‚(:Ý ȵêÔœ’ª€ network.recv‚ 2.145331e+06) disk./.usagePercent‚75.84338893276666 disk.rbd2.in‚0 disk.dm-0.out‚ 78045.8359375 disk.md0p1.out‚ 740.41796875 proc.cpu.threads‚17 network.sent‚ 1.4455853e+07 disk.sdb.out‚ 768.392578125 disk.rbd0.in‚0 disk.sdb1.in‚0 disk.sdb2.in‚0.828125 disk.sda1.out‚0( proc.memory.availableMB‚ 1299.7734375) proc.memory.percent‚18.32554340362549 disk.nvme1n1.in‚ 498.80078125 disk.md0.out‚ 740.41796875$ memory_percent‚92.06681251525879# proc.memory.rssMB‚ 3002.45703125 disk.nvme0n1.in‚ 603.4453125 disk.sdb.in‚0.828125 disk.sda2.out‚ 768.392578125 disk.sdb1.out‚0 disk.sdb2.out‚ 768.392578125 disk.sda.in‚ 4.30078125 disk.rbd2.out‚ 1.845703125 disk.dm-0.in‚ 1102.24609375 disk.sda.out‚ 768.392578125 disk.md0.in‚ 5.12890625 cpu‚25.19601408414425$ disk./.usageGB‚2712.526954650879 disk.nvme0n1.out‚ 34625.65625 disk.sda2.in‚ 4.30078125" disk.nvme1n1.out‚ 43420.1796875 disk.sda1.in‚0 disk.md0p1.in‚ 5.12890625 disk.rbd0.out‚ 1.68359375z¤ý2ù- _wandbruntime‚6692  _runtime‚6692ì2`SÇúÁ & lr-AdamW/pg1‚0.0005331030931537278 & lr-AdamW/pg2‚0.0005331030931537278 trainer/global_step‚5799 % _timestamp‚1.7885170674696724e+09 _runtime‚6692.940163921 _step‚472ï‚jûÌûÆ trainer/global_step‚5799 % _timestamp‚1.7885170674696724e+09 _runtime‚6692.940163921 _step‚472 & lr-AdamW/pg1‚0.0005331030931537278 & lr-AdamW/pg2‚0.0005331030931537278Øæ"2ü- _wandbruntime‚6692  _runtime‚6692wÍIS|ýöO QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 119268606 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6555125173238 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6869986830848 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 _runtime‚6692.941612698 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410816 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 145251229 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚464848168663568 Js€sMonitor.on_train_batch_start/oversize_allocations.current‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 % _timestamp‚1.7885170674709048e+09 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚481960755836928 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚475381097541120 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450780 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 trainer/global_step‚5799 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407872 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6578758887936 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26088 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚538914802352640 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 230957056 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 _step‚473 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 108836525 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚74165733 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 254897152 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚532045070418944 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚475381097541120 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955591168 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411420 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6554227761818 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 398533632 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007241 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 119268606 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955591168 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚532045046478848 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177047040 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚74165735 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214665 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 145251229 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 119268608 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 264519837 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚624 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚34670792 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚471402396425386 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚481959838426112 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 108837151 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1212 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6579658295808 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚471403310876166 FI(þûO WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 264519837 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚624 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚34670792 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚471402396425386 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚481959838426112 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 108837151 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1212 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6579658295808 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚471403310876166 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 119268606 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6555125173238 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6869986830848 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 _runtime‚6692.941612698 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410816 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 145251229 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚464848168663568 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177047040 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚74165735 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214665 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 145251229 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 119268608 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712ÙrWÖ2ÿ- _wandbruntime‚6694  _runtime‚6694Ùë¦'€€N L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 108854599 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6580771112448 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 119288506 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6555336423136 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6870935399936 % _timestamp‚1.7885170686540084e+09 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚539005537108992 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚539005301099520 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 398533632 _runtime‚6694.126648274 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚622 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚482040581524992 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 264564352 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 264564352 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177047040 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007241 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899421184 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚464926148757696 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 236009472 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚482041498949120 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚475460727836672 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 108855221 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1187 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6871147469312 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚471482399631820 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6580771112448 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 264538236 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6579871691264 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚475460727836672 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31686 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 119288506 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 145275846 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26114 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955591168 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411628 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 264538236ˆ™±$…N UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6579871691264 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚74178109 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚532134365699584 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 119288504 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 145249732 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚482041498949120 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚475460709833728 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 145275846 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 236009472 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚482041498949120 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚475460727836672 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 108855221 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1187 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6871147469312 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚471482399631820 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6580771112448 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 264538236 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6579871691264 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚475460727836672 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31686 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 119288506 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 145275846 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26114 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955591168 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411628 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 264538236 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 108854599 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6580771112448 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 119288506 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6555336423136 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6870935399936 % _timestamp‚1.7885170686540084e+09 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚539005537108992 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚539005301099520 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 398533632 _runtime‚6694.126648274 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚622 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚482040581524992 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 264564352 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.all)2Ò»çocated‚ 264564352 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177047040 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007241 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899421184 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚464926148757696 FDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _runtime‚6746.573593668 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 146484131 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6928150589952 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 266721549 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚485997007783936 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚641 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 146484131 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 120263506 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚34959102 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚536511031455744 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450780 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚468747173369968 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚479362594315776 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 120263508 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚543438931342848 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607©U¡<(úO \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚479362594315776 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 120263508 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚543438931342848 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚485997007783936 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1187 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _runtime‚6746.573593668 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 146484131 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6928150589952 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 266721549 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚485997007783936 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚641 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 146484131 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 120263506 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚34959102 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚536511031455744 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450780 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚468747173369968 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712Ý…uAY2‘- _wandbruntime‚6747  _runtime‚6747w÷|‡’€N ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚479442242614272 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 146508748 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚vz‘%475436850995328 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚479442242614272 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26114 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 % _timestamp‚1.7885171221614482e+09 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 109761824 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1187 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 398533632 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚468825170503456 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177047040 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31686 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26114 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚543529417872384 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6611680491872 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6610783080244 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 trainer/global_step‚5849 _step‚478 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 266792154 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚536600326736384 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6635526271488 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚74796909 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899421184 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚74796911 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚486078668306944 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6636425692672 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450988 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 266792154 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 120283404 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚468825153464096 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 146482634 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚623 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚34964915 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 120283404 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚479442224611328 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917424128 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚479442224611328 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 109761824 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1187 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 398533632 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚468825170503456 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177047040 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31686 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26114 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚543529417872384 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6611680491872 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6610783080244 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 trainer/global_step‚5849 _step‚478 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 266792154 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚536600326736384 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6635526271488 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚74796909 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899421184 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚74796911 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚486078668306944 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6636425692672 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450988 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 266792154 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 120283404 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚468825153464096 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 146482634 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚623 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚34964915 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 120283404 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚479442224611328 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917424128 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚479442224611328 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 % _timestamp‚1.7885171753392425e+09 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 121258408 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450780 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚483344127096320 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6690967456256 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 trainer/global_step‚5899 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6690068048384 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚621 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410816 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407872 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚472646195115728 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚472646178076368 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177047040 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 147690945 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚49003”žÞ`Ê5094552576 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917410816 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6690068048384 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚35246859 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 254897152 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 110650813 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 110650192 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 121258408 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚490034177141760 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407872 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 _runtime‚6800.809702923 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 121258406 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 147690945 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚547963439598592 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚490034177141760 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚483344127096320 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 121258406 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚35247478 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1187 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 268949351 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚75403333 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 230957056 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6690967456256 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 398533632 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚479311299152402 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712¡”þ(¤ûO R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚490034177141760 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407872 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 _runtime‚6800.809702923 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 121258406 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 147690945 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚547963439598592 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚490034177141760 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚483344127096320 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 121258406 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚35247478 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1187 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 268949351 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚75403333 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 230957056 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6690967456256 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 398533632 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚479311299152402 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 % _timestamp‚1.7885171753392425e+09 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 121258408 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450780 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚483344127096320 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6690967456256 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 trainer/global_step‚5899 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6690068048384 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚621 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410816 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407872 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚472646195115728 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚472646178076368 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177047040 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 147690945 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚490035094552576 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917410816 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6690068048384 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚35246859 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 254897152 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 110650813 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 110650192 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 121258408 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚472724175209856 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚479391302358836 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955591168 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚35253250 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6691180851712 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 121278304 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31686 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚479390387907848 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚483423757391872 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 110668342 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007241 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚6801.784791683 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6692080272896 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6987605841920 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 147741650 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 398533632 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 268993840 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚472724158170496 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 121278304 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 269019956 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 110668961 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚548053917555200 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6667127148980 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411628 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚490115837664768 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚483423757391872 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6692080272896 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1187 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚75415709 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚617 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚35252633 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 147715536 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚490115837664768 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚619 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚541066287773184 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 268993840 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚483423739388928 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 147741650 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 398533632 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 268993840 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚472724158170496 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 121278304 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 269019956 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 110668961 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚548053917555200 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6667127148980 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411628 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚490115837664768 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚483423757391872 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6692080272896 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1187 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚75415709 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚617 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚35252633 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 147715536 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚490115837664768 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚619 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚541066287773184 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 268993840 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚483423739388928 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚472724175209856 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚479391302358836 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955591168 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚35253250 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6691180851712 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 121278304 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31686 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚479390387907848 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚483423757391872 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 110668342 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007241 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚6801.784791683 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6692080272896 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6987605841920 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚494071346499584 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1212 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 122253308 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 398533632 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6746622036480 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6745722628608 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 122253306 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚494071346499584 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 209985536 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411420 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 111556264 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 148923847 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚487325641873920 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚487325623870976 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177047040 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 271203243 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26088 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚628 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955591168 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚626 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 122253306 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 271177153 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚35534131 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _step‚485 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450780`›b(¶ûO ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 209985536 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411420 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 111556264 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 148923847 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚487325641873920 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚487325623870976 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177047040 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 271203243 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26088 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚628 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955591168 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚626 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 122253306 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 271177153 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚35534131 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _step‚485 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450780 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407872 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚545442929589248 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚483266664966690 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚545442953529344 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26090 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 148923847 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚476545199822128 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚35534757 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 % _timestamp‚1.7885172285594277e+09 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚494072263910400 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 122253308 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚494071346499584 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1212 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 122253308 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 398533632 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6746622036480 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6745722628608 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 122253306 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚494071346499584 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177047040åÔÚA2·- _wandbruntime‚6854  _runtime‚6854]t~M¸€N ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.fry‹C¹&eed‚487405254166528 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6747734853120 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6746835431936 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 224652288 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 122273204 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 _step‚486 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 122273206 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚35540487 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚552577844887040 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 148948438 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚630 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚628 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 398533632 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚545532272750080 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚76034511 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 111574998 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26114 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚476623179916256 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 271221642 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚476623162876896 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6721676394460 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007241 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6746835431936 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450988 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 % _timestamp‚1.7885172295070362e+09 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955591168 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚494152089598464 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 271247758 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7045820729344 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚494153007022592 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 122273206 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 148974552 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 148974552 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 _runtime‚6854.977849446 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917424128 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411628 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899421184 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚487405272169472 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚545532248809984 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177047040 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚487405272169472 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚35539859 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 248592384 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 271247758 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177047040 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 111574368 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚494153007022592 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26114 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚487405254166528 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 148948438 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6747734853120 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚483345753722344 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7045596077056 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 122273204 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6722573806088 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712^ˆ Ê '¹…N H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 271247758 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177047040 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 111574368 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚494153007022592 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26114 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚487405254166528 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 148948438 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6747734853120 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚483345753722344 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7045596077056 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 122273204 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6722573806088 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚487405254166528 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6747734853120 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6746835431936 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 224652288 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 122273204 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 _step‚486 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 122273206 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚35540487 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚552577844887040 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 148948438 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚630 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚628 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 398533632 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚545532272750080 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚76034511 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 111574998 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26114 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚476623179916256 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 271221642 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚476623162876896 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6721676394460 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007241 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6746835431936 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450988 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 % _timestamp‚1.7885172295070362e+09 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955591168 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚494152089598464 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 271247758 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7045820729344 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚494153007022592 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 122273206 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 148974552 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 148974552 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 _runtime‚6854.977849446 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917424128 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411628 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899421184 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚487405272169472 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚545532248809984 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177047040 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚487405272169472 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚35539859 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 248592384 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0æ† –?2º- _wandbruntime‚6854  _runtime‚6854ó•Š4]»× /loss/FRt3/detection_loss‚2.1796823740005493 .loss/FBt1/detection_loss‚2.892633523259844 /loss/FBt2/detection_loss‚0.5427736185025424 1loss/FBt1/assignment_loss‚0.34176127060287137 1loss/FBt2/assignment_loss‚0.10815861341636293 0loss/SRqqt1/detection_loss‚1.828102263998478 &loss/total_loss‚40.469663679561386 trainer/global_step‚5949 $ _timestamp‚1.788517229509224e+09 _runtime‚6854.979349892 1loss/SRqqt3/detection_loss‚1.5059237406988228 1loss/SRqqt4/detection_loss‚2.3663314970640035 .loss/FBt3/detection_loss‚2.012167968920299 0loss/FBt4/assignment_loss‚0.2586880356556531 0loss/FRt3/assignment_loss‚2.5563150525093077 /loss/FBt4/detection_loss‚2.4424445583270145 0loss/FRt4/assignment_loss‚3.0048621230655246 2loss/SRqqt1/assignment_loss‚1.4816341214674584 1loss/FBt3/assignment_loss‚0.31869913830554913 _step‚487 2loss/SRqqt2/assignment_loss‚1.0245073168937648 /loss/FRt4/detection_loss‚3.0802169177267285 0loss/SRqqt2/detection_loss‚1.169691379474742 epoch‚4 0loss/FRt1/assignment_loss‚3.2912528693675993 0loss/FRt2/assignment_loss‚1.4662786783843205 1loss/SRqqt4/assignment_loss‚1.495883965721497 /loss/FRt1/detection_loss‚2.0575865683786874 2loss/SRqqt3/assignment_loss‚1.3747569788351288 /loss/FRt2/detection_loss‚1.6693111049846328Hi>çb¼Ü 0loss/FRt3/assignment_loss‚2.5563150525093077 /loss/FBt4/detection_loss‚2.4424445583270145 0loss/FRt4/assignment_loss‚3.0048621230655246 2loss/SRqqt1/assignment_loss‚1.4816341214674584 1loss/FBt3/assignment_loss‚0.31869913830554913 _step‚487 2loss/SRqqt2/assignment_loss‚1.0245073168937648 /loss/FRt4/detection_loss‚3.0802169177267285 0loss/SRqqt2/detection_loss‚1.169691379474742 epoch‚4 0loss/FRt1/assignment_loss‚3.2912528693675993 0loss/FRt2/assignment_loss‚1.4662786783843205 1loss/SRqqt4/assignment_loss‚1.495883965721497 /loss/FRt1/detection_loss‚2.0575865683786874 2loss/SRqqt3/assignment_loss‚1.3747569788351288 /loss/FRt2/detection_loss‚1.6693111049846328 /loss/FRt3/detection_loss‚2.1796823740005493 .loss/FBt1/detection_loss‚2.892633523259844 /loss/FBt2/detection_loss‚0.5427736185025424 1loss/FBt1/assignment_loss‚0.34176127060287137 1loss/FBt2/assignment_loss‚0.10815861341636293 0loss/SRqqt1/detection_loss‚1.828102263998478 &loss/total_loss‚40.469663679561386 trainer/global_step‚5949 $ _timestamp‚1.788517229509224e+09 _runtime‚6854.979349892 1loss/SRqqt3/detection_loss‚1.5059237406988228 1loss/SRqqt4/detection_loss‚2.3663314970640035 .loss/FBt3/detection_loss‚2.012167968920299 0loss/FBt4/assignment_loss‚0.2586880356556531çíÃOÍ’½‚(:‡ í¶êÔ­ƒ“ÿ gpu.0.gpu‚34.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚37.0 gpu.0.powerWatts‚121.536' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚30.384 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¿vj¾‚(:ß í¶êÔØõ²€ disk.sdb2.in‚ 0.92578125! disk.sdb2.out‚787.62158203125 disk.sdb.out‚787.62158203125 disk.sda.out‚787.62158203125 disk.rbd0.in‚0 disk.sdb1.in‚0$ disk./.usageGB‚2712.812282562256 disk.dm-0.out‚79719.26171875! disk.sda2.out‚787.62158203125 disk.rbd0.out‚ 1.68359375 disk.md0.in‚5.3125 network.recv‚ 2.193874e+06 disk.sda1.out‚0# proc.memory.rssMB‚ 3002.45703125 disk.sda1.in‚0 disk.md0p1.out‚ 758.96875# disk.nvme1n1.out‚44602.78515625) disk./.usagePercent‚75.85136682058777 disk.nvme1n1.in‚ 498.80078125$ memory_percent‚92.06454753875732 network.sent‚ 1.483536e+07" disk.nvme0n1.out‚ 35116.4765625 disk.rbd2.in‚0 disk.dm-0.in‚ 1102.390625 disk.sdb.in‚ 0.92578125) proc.memory.availableMB‚ 1300.14453125) proc.memory.percent‚18.32554340362549 cpu‚25.191712589700185 proc.cpu.threads‚17 disk.md0p1.in‚5.3125 disk.nvme0n1.in‚ 603.58984375 disk.sda.in‚ 4.38671875 disk.sda2.in‚ 4.38671875 disk.sdb1.out‚0 disk.md0.out‚ 758.96875 disk.rbd2.out‚ 1.845703125tTd”¿‚(:‰ ü¶êÔ¹Ö¼ÿ gpu.0.gpu‚46.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚164.649' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚41.16225 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ja¹`À‚(:Õ ü¶êÔÙŒé€ disk.rbd0.out‚ 1.68359375# disk.nvme1n1.out‚44602.98828125 disk.md0.in‚5.3125 disk.sdb2.out‚789.2001953125 disk.sda1.out‚0 disk.dm-0.in‚ 1102.390625 disk.dm-0.out‚79719.5) proc.memory.percent‚18.32554340362549 proc.cpu.threads‚17 disk.rbd2.in‚0 disk.sdb2.in‚ 0.92578125 disk.sda1.in‚0 disk.sda.in‚ 4.38671875 disk.sdb1.out‚0 disk.rbd2.out‚ 1.845703125 disk.md0p1.out‚ 760.484375 disk.sdb.in‚ 0.92578125 disk.md0.out‚ 760.484375 disk.sda2.out‚789.2001953125$ memory_percent‚92.06652641296387 cpu‚25.19121988012936 disk.nvme1n1.in‚ 498.80078125 disk.sdb1.in‚0 disk.sdb.out‚789.2001953125 disk.sda.out‚789.2001953125 network.recv‚ 2.19932e+06( proc.memory.availableMB‚ 1299.8203125 network.sent‚ 1.4932371e+07$ disk./.usageGB‚2712.812213897705 disk.md0p1.in‚5.3125 disk.nvme0n1.in‚ 603.58984375# disk.nvme0n1.out‚35116.51171875 disk.sda2.in‚ 4.38671875# proc.memory.rssMB‚ 3002.45703125) disk./.usagePercent‚75.85136490069819 disk.rbd0.in‚0³®ÒÁ‚(:… ‹·êÔ–Ô¦ÿ gpu.0.gpu‚52.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚37.0 gpu.0.powerWatts‚125.88' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚31.47 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ïOÛf‚(:Û ‹·êÔ¹˜š€ disk.sdb1.out‚0 disk.md0.in‚5.3125 disk.rbd2.out‚ 1.845703125 disk.sda1.out‚0 disk.nvme0n1.in‚ 603.58984375 disk.nvme1n1.in‚ 498.80078125$ memory_percent‚92.06655025482178 network.recv‚ 2.203646e+06# proc.memory.rssMB‚ 3002.45703125% disk./.usageGB‚2712.7940826416016 disk.sda1.in‚0 disk.md0p1.in‚5.3125 disk.sda.in‚ 4.38671875 cpu‚25.190809433881892 disk.md0.out‚ 761.87890625) proc.memory.percent‚18.32554340362549 proc.cpu.threads‚17 disk.dm-0.out‚ 79845.0546875 disk.sda.out‚ 790.65234375# disk.nvme1n1.out‚44604.37109375 disk.sdb2.out‚ 790.65234375 disk.rbd2.in‚0 network.sent‚ 1.4938493e+07 disk.dm-0.in‚ 1102.390625 disk.md0p1.out‚ 761.87890625# disk.nvme0n1.out‚35240.68359375 disk.sdb.in‚ 0.92578125 disk.sdb.out‚ 790.65234375 disk.sda2.out‚ 790.65234375 disk.rbd0.out‚ 1.68359375 disk.sdb1.in‚0 disk.sdb2.in‚ 0.92578125) proc.memory.availableMB‚ 1299.81640625) disk./.usagePercent‚75.85085794318903 disk.sda2.in‚ 4.38671875 disk.rbd0.in‚0ÈÜï(”Â(:‰ š·êÔùÀ«ÿ gpu.0.gpu‚12.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚176.461' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚44.11525 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0}iý¯mÄ‚(:â š·êÔËÖÿÿ disk.dm-0.in‚ 1102.390625 disk.sdb.in‚ 0.92578125 network.sent‚ 1.4944802e+07 disk.nvme0n1.in‚ 603.58984375 disk.sda.in‚ 4.38671875 disk.sda2.out‚792.6513671875% disk./.usageGB‚2712.8628730773926 disk.md0.out‚ 763.80859375 disk.sda1.out‚0 disk.md0p1.out‚ 763.80859375# disk.nvme0n1.out‚35402.02734375 disk.sdb.out‚792.6513671875) proc.memory.availableMB‚ 1299.82421875# proc.memory.rssMB‚ 3002.45703125) disk./.usagePercent‚75.85278135256476 disk.sdb1.out‚0 disk.rbd2.in‚0 disk.sdb2.in‚ 0.92578125 disk.md0p1.in‚5.3125) proc.memory.percent‚18.32554340362549 network.recv‚ 2.208514e+06 disk.rbd0.out‚ 1.68359375 disk.sdb1.in‚0 disk.dm-0.out‚80006.55078125 disk.sda.out‚792.6513671875 cpu‚25.19043385027824 proc.cpu.threads‚17 disk.rbd0.in‚0" disk.nvme1n1.out‚ 44604.5234375 disk.nvme1n1.in‚ 498.80078125 disk.md0.in‚5.3125 disk.rbd2.out‚ 1.845703125 disk.sdb2.out‚792.6513671875 disk.sda1.in‚0 disk.sda2.in‚ 4.38671875$ memory_percent‚92.06650257110596Å™øÅ2Å- _wandbruntime‚6908  _runtime‚6908û©b÷ÇÆÁ & lr-AdamW/pg1‚0.0005089217779734685 & lr-AdamW/pg2‚0.0005089217779734685 trainer/global_step‚5999 % _timestamp‚1.7885172827730076e+09 _runtime‚6908.243510899 _step‚488xÇÌÇÆ trainer/global_step‚5999 % _timestamp‚1.7885172827730076e+09 _runtime‚6908.243510899 _step‚488 & lr-AdamW/pg1‚0.0005089217779734685 & lr-AdamW/pg2‚0.0005089217779734685èGm´»2È- _wandbruntime‚6908  _runtime‚6908‰£Ü¸ï ÉöO QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410816 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚557011669119488 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚498109433268224 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955591168 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚35822115 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7103009450496 _runtime‚6908.245387716 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6801377208832 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7102778493440 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚35822757 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 123248206 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6802276616704 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 _step‚489 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚491307156651520 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 123248208 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚480444204528528 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚498108515857408 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 112463692 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411420 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407872 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚549908914566144 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 112463048 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1187 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚487221116330198 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955591168 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚557011924016640 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214665 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚76640933 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26090 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚498108515857408 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177047040 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 123248208 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6801377208832 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚549908890626048 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚480444187489168 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚491307156651520 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 150182837 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 398533632 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚487220201879418 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 150182837 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26088 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917410816 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚491307138648576 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚491307138648576 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1212 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411420 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407872 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚549908914566144 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 112463048 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1187 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚487221116330198 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955591168 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚557011924016640 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214665 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚76640933 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26090 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚498108515857408 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177047040 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 123248208 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6801377208832 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚549908890626048 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚480444187489168 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚491307156651520 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 150182837 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 398533632 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚487220201879418 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 150182837 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26088 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917410816 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚491307138648576 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚491307138648576 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1212 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 _step‚489 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚491307156651520 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 123248208 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚480444204528528 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚498108515857408 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 112463692 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450988 _step‚490 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 123268106 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚491386768944128 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚491386768944128 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 112481160 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚35828491 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚549998233786880 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 123268106 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚480522184622656 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 123268104 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚76653311 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6802490012160 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚642 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚557102391323648 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 273475560 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚491386786947072 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 226749440 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚498190176380416 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 250689536 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26114 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177047040 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 273475560 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 273449444 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6803389433344 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917424128 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 150207454 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31686 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7104157536768 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚498190176380416 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899421184 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899421184 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚498189258956288 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6802490012160 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚487299290634864 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 123268104 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677²D–>—Í…N I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26114 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177047040 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 273475560 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 273449444 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6803389433344 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917424128 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 150207454 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31686 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7104157536768 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚498190176380416 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899421184 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899421184 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚498189258956288 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.|[Kƒtsmall_pool.freed‚ 6802490012160 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚487299290634864 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 123268104 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450988 _step‚490 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 123268106 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚491386768944128 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚491386768944128 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 112481160 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚35828491 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚549998233786880 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 123268106 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚480522184622656 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 123268104 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚76653311 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6802490012160 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚642 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚557102391323648 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 273475560 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚491386786947072 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 226749440 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚498190176380416 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 250689536êˆæ<2Î- _wandbruntime‚6909  _runtime‚6909/B!_ÏÙ 1loss/SRqqt1/detection_loss‚1.4197401230962112 1loss/SRqqt2/detection_loss‚1.0251457963524193 /loss/FBt3/detection_loss‚2.6640302762389183 2loss/SRqqt4/assignment_loss‚1.6387536625923782 1loss/FBt2/assignment_loss‚0.06679310185677734 1loss/FBt4/assignment_loss‚0.09180803598170921 /loss/FRt3/detection_loss‚2.0654902722762554 /loss/FRt4/detection_loss‚3.1687540493228217 _runtime‚6909.428503419 trainer/global_step‚5999 _step‚491 /loss/FRt1/assignment_loss‚2.423975366133231 0loss/FRt3/assignment_loss‚2.2764237218973586 1loss/FBt1/assignment_loss‚0.09480417740996927 /loss/FBt1/detection_loss‚2.5003810950687955 &loss/total_loss‚39.248369335421664 epoch‚4 0loss/FRt2/assignment_loss‚1.4386901248384405 /loss/FBt3/assignment_loss‚0.700139405584423 /loss/FRt2/detection_loss‚1.5453210499965482 1loss/SRqqt3/detection_loss‚1.4940275119192767 1loss/SRqqt4/detection_loss‚2.7824758640651046 .loss/FRt1/detection_loss‚2.007412326749903 /loss/FBt2/detection_loss‚0.6307947046365844 /loss/FBt4/detection_loss‚2.1925502916177115 2loss/SRqqt2/assignment_loss‚0.8966042149891266 $ _timestamp‚1.788517283958207e+09 0loss/FRt4/assignment_loss‚3.9427206218242645 2loss/SRqqt1/assignment_loss‚0.9418084221763601 2loss/SRqqt3/assignment_loss‚1.2397251187970764sþÓAdÐÞ 1loss/SRqqt1/detection_loss‚1.4197401230962112 1loss/SRqqt2/detection_loss‚1.0251457963524193 /loss/FBt3/detection_loss‚2.6640302762389183 2loss/SRqqt4/assignment_loss‚1.6387536625923782 1loss/FBt2/assignment_loss‚0.06679310185677734 1loss/FBt4/assignment_loss‚0.09180803598170921 /loss/FRt3/detection_loss‚2.0654902722762554 /loss/FRt4/detection_loss‚3.1687540493228217 _runtime‚6909.428503419 trainer/global_step‚5999 _step‚491 /loss/FRt1/assignment_loss‚2.423975366133231 0loss/FRt3/assignment_loss‚2.2764237218973586 1loss/FBt1/assignment_loss‚0.09480417740996927 /loss/FBt1/detection_loss‚2.5003810950687955 &loss/total_loss‚39.248369335421664 epoch‚4 0loss/FRt2/assignment_loss‚1.4386901248384405 /loss/FBt3/assignment_loss‚0.700139405584423 /loss/FRt2/detection_loss‚1.5453210499965482 1loss/SRqqt3/detection_loss‚1.4940275119192767 1loss/SRqqt4/detection_loss‚2.7824758640651046 .loss/FRt1/detection_loss‚2.007412326749903 /loss/FBt2/detection_loss‚0.6307947046365844 /loss/FBt4/detection_loss‚2.1925502916177115 2loss/SRqqt2/assignment_loss‚0.8966042149891266 $ _timestamp‚1.788517283958207e+09 0loss/FRt4/assignment_loss‚3.9427206218242645 2loss/SRqqt1/assignment_loss‚0.9418084221763601 2loss/SRqqt3/assignment_loss‚1.2397251187970764ëC(íü’Ñ‚(:‡ ©·êÔô®‚€ gpu.0.gpu‚37.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚37.0 gpu.0.powerWatts‚114.408' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.602 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÏRs“nÒ‚(:ã ©·êԌ̀$ disk./.usageGB‚2712.867012023926 disk.dm-0.out‚80083.12109375 disk.nvme0n1.in‚ 603.58984375 disk.sdb1.in‚0! disk.sda2.out‚794.11669921875! disk.nvme1n1.out‚ 44604.734375 disk.rbd0.in‚0 disk.nvme1n1.in‚ 498.80078125 disk.md0.out‚ 765.21484375) disk./.usagePercent‚75.85289707924213 disk.sda.in‚ 4.38671875 disk.sdb2.in‚ 0.92578125 disk.dm-0.in‚ 1102.390625 disk.md0p1.out‚ 765.21484375 disk.sdb.in‚ 0.92578125 disk.sda.out‚794.11669921875 disk.rbd0.out‚ 1.68359375$ memory_percent‚92.06438064575195 network.recv‚ 2.211951e+06 disk.rbd2.out‚ 1.845703125 disk.sda1.out‚0# disk.nvme0n1.out‚35478.38671875 disk.sdb1.out‚0 cpu‚25.190017686847014 proc.cpu.threads‚17 network.sent‚ 1.4947856e+07 disk.rbd2.in‚0 disk.sda1.in‚0 disk.md0.in‚5.3125# proc.memory.rssMB‚ 3002.45703125! disk.sdb2.out‚794.11669921875 disk.md0p1.in‚5.3125 disk.sdb.out‚794.11669921875 disk.sda2.in‚ 4.38671875' proc.memory.availableMB‚ 1300.171875) proc.memory.percent‚18.32554340362549}TüÃ’Ó‚(:‡ ¸·êÔ´œÿ gpu.0.gpu‚24.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚180.196' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚45.049 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ZÖvfÔ‚(:Û ¸·êÔ§ Ýÿ disk.md0p1.out‚ 766.53125 disk.sdb.out‚795.4970703125 disk.md0.in‚5.3125 disk.md0.out‚ 766.53125# proc.memory.rssMB‚ 3002.45703125 cpu‚25.189614482305274 disk.sda.in‚ 4.38671875# memory_percent‚92.0658826828003 network.recv‚ 2.217722e+06 disk.sdb2.out‚795.4970703125) disk./.usagePercent‚75.85464876516116$ disk./.usageGB‚2712.929660797119 disk.md0p1.in‚5.3125 disk.sdb.in‚ 0.92578125 disk.sda2.in‚ 4.38671875" disk.nvme1n1.out‚ 44604.9296875 disk.rbd2.in‚0 disk.rbd0.out‚ 1.68359375 disk.sdb2.in‚ 0.92578125 disk.sda1.in‚0 disk.sda2.out‚795.4970703125 disk.nvme1n1.in‚ 498.80078125 disk.dm-0.in‚ 1102.390625) proc.memory.percent‚18.32554340362549 proc.cpu.threads‚17 disk.nvme0n1.in‚ 603.58984375# disk.nvme0n1.out‚35550.26953125 disk.rbd0.in‚0 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.sda1.out‚0 disk.dm-0.out‚80155.19921875) proc.memory.availableMB‚ 1299.92578125 network.sent‚ 1.5055691e+07 disk.sda.out‚795.4970703125 disk.rbd2.out‚ 1.845703125‰~,”Õ‚(:‰ Ç·êÔ–ûý gpu.0.gpu‚46.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚37.0 gpu.0.powerWatts‚117.193' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚29.29825 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0“¤YÁkÖ‚(:à Ç·êÔÖÀ»ÿ disk.md0.in‚5.3125 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.sdb2.in‚ 0.92578125 network.sent‚ 1.5061536e+07) disk./.usagePercent‚75.85985603899988 disk.sda1.out‚0 disk.nvme1n1.in‚ 498.80078125 disk.rbd2.out‚ 1.845703125 disk.dm-0.in‚ 1102.390625 disk.sdb.in‚ 0.92578125 disk.sda.out‚797.28955078125 disk.rbd0.out‚ 1.68359375) proc.memory.percent‚18.32554340362549 cpu‚25.18924439299901 proc.cpu.threads‚17! disk.nvme0n1.out‚ 35672.296875 disk.sda2.in‚ 4.38671875# disk.nvme1n1.out‚44605.40234375 disk.rbd2.in‚0$ memory_percent‚92.06733703613281$ disk./.usageGB‚2713.115898132324 disk.sda1.in‚0 disk.md0p1.in‚5.3125 disk.sda.in‚ 4.38671875! disk.sda2.out‚797.28955078125! disk.sdb2.out‚797.28955078125 network.recv‚ 2.221794e+06 disk.dm-0.out‚80277.69921875 disk.md0.out‚ 768.25390625# proc.memory.rssMB‚ 3002.45703125 disk.md0p1.out‚ 768.25390625 disk.nvme0n1.in‚ 603.58984375 disk.rbd0.in‚0% proc.memory.availableMB‚ 1299.6875 disk.sdb.out‚797.28955078125Æ&ë’ׂ(:‡ Ö·êÔþÎÍþ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚36.0 gpu.0.powerWatts‚113.466' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.3665 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0’|N¾iØ‚(:Þ Ö·êÔÏ÷à€ disk.sda1.out‚0 disk.sda.in‚ 4.38671875 disk.sdb1.in‚0 disk.sdb1.out‚0 cpu‚25.19022402813782 network.sent‚ 1.5067805e+07 disk.dm-0.in‚ 1102.390625 disk.sdb.in‚ 0.92578125# disk.nvme1n1.out‚44605.60546875 network.recv‚ 2.226497e+06) disk./.usagePercent‚75.86685115667801 disk.sdb.out‚799.4462890625 disk.md0.in‚5.3125 disk.sda1.in‚0 disk.sda2.in‚ 4.38671875 disk.sda2.out‚799.4462890625 disk.rbd2.in‚0 disk.sdb2.out‚799.4462890625% disk./.usageGB‚2713.3660774230957 disk.sda.out‚799.4462890625# proc.memory.rssMB‚ 3002.45703125) proc.memory.percent‚18.32554340362549 disk.dm-0.out‚ 80463.828125 disk.nvme0n1.in‚ 603.58984375# disk.nvme0n1.out‚35858.22265625 disk.rbd0.in‚0 disk.nvme1n1.in‚ 498.80078125( proc.memory.availableMB‚ 1299.4921875 proc.cpu.threads‚17 disk.md0p1.in‚5.3125 disk.md0p1.out‚ 770.3515625 disk.sdb2.in‚ 0.92578125 disk.rbd0.out‚ 1.68359375 disk.md0.out‚ 770.3515625 disk.rbd2.out‚ 1.845703125$ memory_percent‚92.06852912902832µ®$}”Ù‚(:‰ å·êÔÆ‘Ýÿ gpu.0.gpu‚56.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚173.059' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.26475 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0óþˆ_Ú‚(:Ô å·êÔ¹×Á€ cpu‚25.197210642272548% disk./.usageGB‚2713.3651161193848 disk.md0.out‚ 771.8046875 disk.sda1.out‚0 disk.dm-0.out‚80685.98828125 disk.sda1.in‚0# memory_percent‚92.0684814453125) disk./.usagePercent‚75.86682427822392# disk.nvme1n1.out‚44605.76171875 disk.sdb2.in‚ 0.92578125 disk.md0p1.in‚5.3125" disk.nvme0n1.out‚ 36080.2265625" proc.memory.availableMB‚1299.5 disk.nvme1n1.in‚ 498.80078125 disk.rbd2.in‚0 proc.cpu.threads‚17 network.recv‚ 2.22986e+06 disk.sda.out‚ 800.982421875 disk.sda2.in‚ 4.38671875 disk.rbd0.out‚ 1.68359375 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.sdb.out‚ 800.982421875 disk.rbd0.in‚0 disk.md0.in‚5.3125 disk.sdb2.out‚ 800.982421875 disk.sdb.in‚ 0.92578125 disk.sda.in‚ 4.38671875 disk.sda2.out‚ 800.982421875 disk.rbd2.out‚ 1.845703125 disk.dm-0.in‚ 1102.390625 disk.md0p1.out‚ 771.8046875# proc.memory.rssMB‚ 3002.45703125 network.sent‚ 1.5075256e+07 disk.nvme0n1.in‚ 603.58984375) proc.memory.percent‚18.32554340362549\¼þ®žÛ‚(:“ ô·êÔÙØ‘ gpu.0.gpu‚51.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚37.0 gpu.0.powerWatts‚122.818' gpu.0.enforcedPowerLimitWatts‚400.0) gpu.0.powerPercent‚30.704500000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0“Ruì\Ü‚(:Ñ ô·êÔŽù΂ network.recv‚ 2.234377e+06 disk.dm-0.in‚ 1102.390625# disk.nvme0n1.out‚36080.23046875 disk.sda.out‚ 802.552734375( proc.memory.availableMB‚ 1299.5078125 disk.dm-0.out‚ 80686.1875 disk.md0p1.in‚5.3125 disk.sdb2.in‚ 0.92578125 disk.sda1.in‚0# proc.memory.rssMB‚ 3002.45703125 disk.rbd0.out‚ 1.68359375 disk.sdb1.out‚0 disk.rbd2.in‚0 disk.sdb2.out‚ 802.552734375 network.sent‚ 1.5079031e+07 disk.sda.in‚ 4.38671875 disk.rbd0.in‚0 disk.md0.out‚773.3125 disk.rbd2.out‚ 1.845703125$ memory_percent‚92.06843376159668) proc.memory.percent‚18.32554340362549 proc.cpu.threads‚17 disk.md0p1.out‚773.3125 disk.nvme1n1.in‚ 498.80078125# disk.nvme1n1.out‚44605.95703125 disk.md0.in‚5.3125 disk.sda1.out‚0 cpu‚25.20468432245166$ disk./.usageGB‚2713.365058898926 disk.sdb.out‚ 802.552734375 disk.sda2.in‚ 4.38671875) disk./.usagePercent‚75.86682267831593 disk.nvme0n1.in‚ 603.58984375 disk.sdb.in‚ 0.92578125 disk.sda2.out‚ 802.552734375 disk.sdb1.in‚0[é/’Ý‚(:‡ ƒ¸êÔù¬íÿ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚35.0 gpu.0.powerWatts‚113.466' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.3665 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ue·9oÞ‚(:ä ƒ¸êÔñäÁ€ disk.dm-0.in‚ 1102.390625 disk.dm-0.out‚81094.41796875) proc.memory.percent‚18.32554340362549 proc.cpu.threads‚17 disk.sdb2.in‚ 0.92578125 disk.sda1.out‚0 disk.sda2.in‚ 4.38671875! disk.sda2.out‚804.41259765625 disk.sdb.out‚804.41259765625 disk.sdb1.out‚0' proc.memory.availableMB‚ 1299.734375 disk.md0p1.out‚ 775.11328125# proc.memory.rssMB‚ 3002.45703125 disk.sda.in‚ 4.38671875) disk./.usagePercent‚75.86682267831593 disk.md0p1.in‚5.3125 disk.nvme1n1.in‚ 498.80078125$ memory_percent‚92.06705093383789 disk.sda1.in‚0 disk.rbd2.in‚0 network.sent‚ 1.5087187e+07# disk.nvme0n1.out‚36488.11328125 disk.rbd0.in‚0 disk.md0.in‚5.3125 cpu‚25.212122126786113 disk.sdb.in‚ 0.92578125" disk.nvme1n1.out‚ 44606.3046875 disk.sdb1.in‚0 disk.md0.out‚ 775.11328125 disk.rbd2.out‚ 1.845703125 disk.sda.out‚804.41259765625 network.recv‚ 2.238519e+06$ disk./.usageGB‚2713.365058898926 disk.nvme0n1.in‚ 603.58984375 disk.rbd0.out‚ 1.68359375! disk.sdb2.out‚804.41259765625P[õÑ2ß- _wandbruntime‚7017  _runtime‚7017D çc*àÝT PFDeviceStatsMonitor.on_validation_batch_start/active.small_pool.current‚20519 ZLDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.current‚ 675116544 J@DeviceStatsMonitor.on_validation_batch_start/allocation.all.peak‚31931 KBDeviceStatsMonitor.on_validation_batch_start/segment.all.allocated‚1171 KEDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.freed‚0 MCDeviceStatsMonitor.on_validation_batch_start/active.small_pool.peak‚31686 \LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 dRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.freed‚ 7180647544320 [LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.peak‚ 1175007241 OGDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.current‚564 ZLDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.allocated‚ 151745628 ]NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.current‚ 1182793728 `NDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.allocated‚ 6877430126592 SEDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.freed‚ 114023174 BDeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 QKDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.allocated‚0 NHDeviceStatsMonitor.on_validation_batch_start/oversize_segments.allocated‚0 aMDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.freed‚486470006220432 VHDeviceStatsMonitor.on_validation_batch_start/active.large_pool.allocated‚ 124912040 ZLDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.allocated‚ 124912040 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 OIDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.current‚0 TDDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.peak‚ 14220787712 cODeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.allocated‚563915415262208 B;DeviceStatsMonitor.on_validation_batch_start/max_split_size‚-1 UEDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.peak‚ 13510214665 YIDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.allocated‚ 14220787712 OADeviceStatsMonitor.on_validation_batch_start/allocation.all.freed‚ 276637147 GADeviceStatsMonitor.on_validation_batch_start/num_sync_all_streams‚2 ]ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.current‚ 673144760 ZKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 QIDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.allocated‚607 FDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 OGDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.current‚607 bNDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.allocated‚497595142759936 jVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.allocated‚556734622848000 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.freed‚0 fRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.freed‚556734598907904 LFDeviceStatsMonitor.on_validation_batch_start/active.large_pool.current‚2 LCDeviceStatsMonitor.on_validation_batch_start/active.large_pool.peak‚3677 WIDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.allocated‚ 114023375 \LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.peak‚ 12392107008 QIDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.allocated‚564 I?DeviceStatsMonitor.on_validation_batch_start/active.all.current‚20521 SEDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.current‚ 693119488 _MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.freed‚ 6876755010048 bTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.current‚ 144869888 OGDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.current‚201 RKDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.peak‚35 hVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.allocated‚ 7180792414208 cQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.allocated‚ 6851795427802 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.freed‚0 aMDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.freed‚497595124756992 VHDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.freed‚ 124912038 PGDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.peak‚3677 WCDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.freed‚504471879767040 \JDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.freed‚ 6876755010048 TKDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.peak‚1187 _KDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.freed‚563915246452224 JDDeviceStatsMonitor.on_validation_batch_start/oversize_segments.freed‚0 PJDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.current‚2 K=DeviceStatsMonitor.on_validation_batch_start/active.all.freed‚ 276637147 XIDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.peak‚ 1177047040 eQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.allocated‚497595142759936 aTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.current‚23940096 % _timestamp‚1.7885173917420893e+09 YLDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.current‚18002944 trainer/global_step‚6044 ZFDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.freed‚493321128503474 RBDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.peak‚ 13955591168 [KDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.freed‚77672789 E?DeviceStatsMonitor.on_validation_batch_start/num_oom_rejections‚0 I@DeviceStatsMonitor.on_validation_batch_start/segment.all.current‚1171 SEDeviceStatsMonitor.on_validation_batch_start/allocation.all.allocated‚ 276657668 LFDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.freed‚0 [GDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.allocated‚504472572886528 [LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 VHDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.freed‚ 151725109 F=DeviceStatsMonitor.on_validation_batch_start/num_device_alloc‚1171 LFDeviceStatsMonitor.on_validation_batch_start/oversize_segments.current‚0 WGDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.current‚ 14220787712 LDDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.peak‚564 TNDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.current‚2 VHDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.current‚ 693119488zV¡yáâT D>DeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 QKDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.allocated‚0 NHDeviceStatsMonitor.on_validation_batch_start/oversize_segments.allocated‚0 aMDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.freed‚486470006220432 VHDeviceStatsMonitor.on_validation_batch_start/active.large_pool.allocated‚ 124912040 ZLDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.allocated‚ 124912040 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 OIDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.current‚0 TDDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.peak‚ 14220787712 cODeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.allocated‚563915415262208 B;DeviceStatsMonitor.on_validation_batch_start/max_split_size‚-1 UEDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.peak‚ 13510214665 YIDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.allocated‚ 14220787712 OADeviceStatsMonitor.on_validation_batch_start/allocation.all.freed‚ 276637147 GADeviceStatsMonitor.on_validation_batch_start/num_sync_all_streams‚2 ]ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.current‚ 673144760 ZKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 QIDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.allocated‚607 FDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 OGDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.current‚607 bNDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.allocated‚497595142759­O«ÄÙ936 jVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.allocated‚556734622848000 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.freed‚0 fRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.freed‚556734598907904 LFDeviceStatsMonitor.on_validation_batch_start/active.large_pool.current‚2 LCDeviceStatsMonitor.on_validation_batch_start/active.large_pool.peak‚3677 WIDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.allocated‚ 114023375 \LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.peak‚ 12392107008 QIDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.allocated‚564 I?DeviceStatsMonitor.on_validation_batch_start/active.all.current‚20521 SEDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.current‚ 693119488 _MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.freed‚ 6876755010048 bTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.current‚ 144869888 OGDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.current‚201 RKDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.peak‚35 hVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.allocated‚ 7180792414208 cQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.allocated‚ 6851795427802 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.freed‚0 aMDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.freed‚497595124756992 VHDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.freed‚ 124912038 PGDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.peak‚3677 WCDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.freed‚504471879767040 \JDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.freed‚ 6876755010048 TKDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.peak‚1187 _KDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.freed‚563915246452224 JDDeviceStatsMonitor.on_validation_batch_start/oversize_segments.freed‚0 PJDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.current‚2 K=DeviceStatsMonitor.on_validation_batch_start/active.all.freed‚ 276637147 XIDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.peak‚ 1177047040 eQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.allocated‚497595142759936 aTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.current‚23940096 % _timestamp‚1.7885173917420893e+09 YLDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.current‚18002944 trainer/global_step‚6044 ZFDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.freed‚493321128503474 RBDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.peak‚ 13955591168 [KDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.freed‚77672789 E?DeviceStatsMonitor.on_validation_batch_start/num_oom_rejections‚0 I@DeviceStatsMonitor.on_validation_batch_start/segment.all.current‚1171 SEDeviceStatsMonitor.on_validation_batch_start/allocation.all.allocated‚ 276657668 LFDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.freed‚0 [GDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.allocated‚504472572886528 [LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 VHDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.freed‚ 151725109 F=DeviceStatsMonitor.on_validation_batch_start/num_device_alloc‚1171 LFDeviceStatsMonitor.on_validation_batch_start/oversize_segments.current‚0 WGDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.current‚ 14220787712 LDDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.peak‚564 TNDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.current‚2 VHDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.current‚ 693119488 PFDeviceStatsMonitor.on_validation_batch_start/active.small_pool.current‚20519 ZLDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.current‚ 675116544 J@DeviceStatsMonitor.on_validation_batch_start/allocation.all.peak‚31931 KBDeviceStatsMonitor.on_validation_batch_start/segment.all.allocated‚1171 KEDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.freed‚0 MCDeviceStatsMonitor.on_validation_batch_start/active.small_pool.peak‚31686 \LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 dRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.freed‚ 7180647544320 [LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.peak‚ 1175007241 OGDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.current‚564 ZLDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.allocated‚ 151745628 ]NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.current‚ 1182793728 `NDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.allocated‚ 6877430126592 SEDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.freed‚ 114023174 BDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚1171 _runtime‚7018.314820624 G=DeviceStatsMonitor.on_validation_batch_end/active.all.current‚20521 ZMDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.current‚18002944 ]IDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.freed‚563971184486912 aODeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.allocated‚ 6852282354526 MEDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.current‚201 KADeviceStatsMonitor.on_validation_batch_end/active.small_pool.peak‚31686 RIDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.peak‚1187 BDeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚31931 NHDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.current‚2 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.allocated‚36355863 VHDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.peak‚ 398533632 OIDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.allocated‚0 D:DeviceStatsMonitor.on_validation_batch_end/active.all.peak‚31931 YEDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.allocated‚504526632724992 YKDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.current‚ 168809984 JDDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.peak‚0 JDDeviceStatsMonitor.on_validation_batch_end/oversize_segments.current‚0 ]KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.freed‚ 6851609209766 YJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.peak‚ 1175007241 PBDeviceStatsMonitor.on_validation_batch_end/active.small_pool.freed‚ 151733883 `LDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.allocated‚497648714035712 trainer/global_step‚6044 XDDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.freed‚504525939605504 SCDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.peak‚ 13955591168 bPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.freed‚ 7181136107008 JDDeviceStatsMonitor.on_validation_batch_end/active.large_pool.current‚2 TFDeviceStatsMonitor.on_validation_batch_end/active.small_pool.allocated‚ 151754402 aODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.allocated‚ 6877918689280 @9DeviceStatsMonitor.on_validation_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_validation_batch_end/num_sync_all_streams‚2 SCDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.peak‚ 13510214665 ZHDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.freed‚ 6877243572736 _RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.current‚23940096 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.current‚ 693119488 P@DeviceStatsMonitor.on_validation_batch_end/active_bytes.all.peak‚ 13955591168 RHDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.current‚20519 \HDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.allocated‚493372075704142 TFDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.current‚ 690184120 ZJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.peak‚ 12392107008 % _timestamp‚1.7885173928436928e+09 QCDeviceStatsMonitor.on_validation_batch_end/allocation.all.allocated‚ 276681718 OEDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.peak‚31686 aMDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.allocated‚563971353296896 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 D;DeviceStatsMonitor.on_validation_batch_end/num_device_alloc‚1171 RBDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.peak‚ 14220787712 MEDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.current‚607 TLDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.current‚199 C=DeviceStatsMonitor.on_validation_batch_end/num_oom_rejections‚0 YIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 ICDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.freed‚0 93DeviceStatsMonitor.on_validation_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_validation_batch_end/oversize_segments.peak‚0 JADeviceStatsMonitor.on_validation_batch_end/active.large_pool.peak‚3677 cODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.allocated‚497648714035712 XJDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.allocated‚ 124927316 TFDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.freed‚ 151733883 @:DeviceStatsMonitor.on_validation_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_validation_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚31931 NHDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.current‚2 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.allocated‚36355863 VHDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.peak‚ 398533632 OIDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.allocated‚0 D:DeviceStatsMonitor.on_validation_batch_end/active.all.peak‚31931 YEDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.allocated‚504526632724992 YKDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.current‚ 168809984 JDDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.peak‚0 JDDeviceStatsMonitor.on_validation_batch_end/oversize_segments.current‚0 ]KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.freed‚ 6851609209766 YJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.peak‚ 1175007241 PBDeviceStatsMonitor.on_validation_batch_end/active.small_pool.freed‚ 151733883 `LDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.allocated‚497648714035712 trainer/global_step‚6044 XDDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.freed‚504525939605504 SCDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.peak‚ 13955591168 bPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.freed‚ 7181136107008 JDDeviceStatsMonitor.on_validation_batch_end/active.large_pool.current‚2 TFDeviceStatsMonitor.on_validation_batch_end/active.small_pool.allocated‚ 151754402 aODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.allocated‚ 6877918689280 @9DeviceStatsMonitor.on_validation_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_validation_batch_end/num_sync_all_streams‚2 SCDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.peak‚ 13510214665 ZHDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.freed‚ 6877243572736 _RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.current‚23940096 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.current‚ 693119488 P@DeviceStatsMonitor.on_validation_batch_end/active_bytes.all.peak‚ 13955591168 RHDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.current‚20519 \HDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.allocated‚493372075704142 TFDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.current‚ 690184120 ZJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.peak‚ 12392107008 % _timestamp‚1.7885173928436928e+09 QCDeviceStatsMonitor.on_validation_batch_end/allocation.all.allocated‚ 276681718 OEDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.peak‚31686 aMDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.allocated‚563971353296896 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 D;DeviceStatsMonitor.on_validation_batch_end/num_device_alloc‚1171 RBDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.peak‚ 14220787712 MEDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.current‚607 TLDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.current‚199 C=DeviceStatsMonitor.on_validation_batch_end/num_oom_rejections‚0 YIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 ICDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.freed‚0 93DeviceStatsMonitor.on_validation_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_validation_batch_end/oversize_segments.peak‚0 JADeviceStatsMonitor.on_validation_batch_end/active.large_pool.peak‚3677 cODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.allocated‚497648714035712 XJDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.allocated‚ 124927316 TFDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.freed‚ 151733883 @:DeviceStatsMonitor.on_validation_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_validation_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚1171 _runtime‚7018.314820624íö&€”å‚(:‰ ’¸êÔÒƹƒ gpu.0.gpu‚61.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚37.0 gpu.0.powerWatts‚182.641' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚45.66025 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÿdk§oæ‚(:ä ’¸êÔ”ö¯† cpu‚25.219326638100537 disk.dm-0.out‚81358.55078125 disk.md0p1.in‚5.3125! disk.sda2.out‚806.08642578125# proc.memory.rssMB‚ 3002.45703125) proc.memory.percent‚18.32554340362549 proc.cpu.threads‚17 disk.nvme0n1.in‚ 603.58984375 disk.md0p1.out‚ 776.72265625 disk.sda.out‚806.08642578125 disk.rbd2.out‚ 1.845703125! disk.sdb2.out‚806.08642578125 network.sent‚ 1.5104207e+07# disk.nvme1n1.out‚44606.45703125 disk.md0.in‚5.3125 disk.rbd2.in‚0 disk.dm-0.in‚ 1102.390625 disk.rbd0.out‚ 1.68359375( disk./.usagePercent‚75.8668780351321 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.sda1.in‚0 disk.nvme0n1.out‚ 36752.09375 disk.sda2.in‚ 4.38671875 disk.rbd0.in‚0) proc.memory.availableMB‚ 1300.95703125% disk./.usageGB‚2713.3670387268066 disk.md0.out‚ 776.72265625 disk.sdb.out‚806.08642578125 disk.sda.in‚ 4.38671875$ memory_percent‚92.05958843231201 disk.sda1.out‚0 disk.sdb.in‚ 0.92578125 disk.sdb2.in‚ 0.92578125 network.recv‚ 2.243438e+06 disk.nvme1n1.in‚ 498.800781257Ñ2ç- _wandbruntime‚7033  _runtime‚7033¿LÕŸ© è¨ @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 ;"Purity/3FRt0SRqqt*FBt/event_purity‚0.011111111380159855 particle/accuracy_6_of_7‚1 ) Purity/*FRt1SRqqt3FBt/FRt_purity‚0.25 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.5247529149055481 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 +jet/accuracy_2_of_2‚0.11170494556427002 & Purity/1FRt0SRqqt4FBt/FRt_purity‚0 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 7 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.2196120172739029 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9766082167625427 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.4692222774028778 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.10089284926652908 particle/accuracy_1_of_1‚1 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.28549379110336304 9"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.4000000059604645 :"Purity/0FRt3SRqqt0FBt/event_purity‚0.03104575164616108 ,"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.875 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 <"Purity/1FRt2SRqqt0FBt/event_purity‚0.0012210012646391988 9"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.4309942424297333 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 *jet/accuracy_2_of_3‚0.2587764859199524 particle/accuracy_2_of_3‚1 particle/accuracy_3_of_5‚1 :"Purity/*FRt3SRqqt0FBt/event_purity‚0.00994623638689518 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 particle/accuracy_5_of_6‚1 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.9074073433876038 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9931722283363342 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 _runtime‚7033.527134075 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.4059998393058777 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.22158724069595337 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.4555996358394623 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.4270833134651184 *"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.5 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.7142857313156128 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.027457214891910553 :"Purity/0FRt*SRqqt2FBt/event_purity‚0.25389009714126587 , Purity/1FRt0SRqqt3FBt/FRHæÛ½ùt_purity‚0.59375 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.1130499392747879 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.010366199538111687 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 * Purity/4FRt*SRqqt0FBt/FRt_purity‚0.125 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚1 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 -"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.4375 jet/accuracy_6_of_7‚0 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 & Purity/*FRt3SRqqt2FBt/FRt_purity‚1 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.4232521057128906 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 & Purity/*FRt0SRqqt4FBt/FRt_purity‚0 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.33762624859809875 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 & Purity/*FRt*SRqqt4FBt/FRt_purity‚0 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.11347518116235733 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 & Purity/*FRt3SRqqt1FBt/FBt_purity‚1 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.3333333432674408 9"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.2916666567325592 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.04435495287179947 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 *jet/accuracy_2_of_5‚0.4909690022468567 ("Purity/0FRt2SRqqt3FBt/SRqqt_purity‚1 epoch‚4 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9290780425071716 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.22666670382022858 jet/accuracy_5_of_7‚0 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9404761791229248 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.23269513249397278 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.19976213574409485 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 particle/accuracy_3_of_4‚1 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.3611111044883728 9"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.4000000059604645 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.2222222238779068 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 7 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.8571428656578064 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 6 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.274755597114563 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.22949247062206268 8"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.521170437335968 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 <"Purity/3FRt*SRqqt*FBt/event_purity‚0.0023419205099344254 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9900000095367432 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.6040292382240295 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.9122807383537292 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8080624938011169 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.5211611390113831 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.20164605975151062 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 +"Purity/1FRt1SRqqt3FBt/SRqqt_purity‚0.75 :"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.47789621353149414 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.1805555671453476 *"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.5 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.26102006435394287 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 :"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.43924060463905334 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 7 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.3083333373069763 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9938737750053406 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 jet/accuracy_1_of_7‚1 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9990079402923584 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5272077322006226 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 8 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.35462963581085205 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.19696968793869019 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.5170079469680786 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.5196301937103271 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.19421087205410004 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.7575758099555969 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.10526315867900848 +jet/accuracy_2_of_6‚0.49358975887298584 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.23135557770729065 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.02986210398375988 ;"Purity/0FRt1SRqqt2FBt/event_purity‚0.056752875447273254 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.02551707811653614 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_4_of_5‚1 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 6 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.998742938041687 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9826388955116272 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.20000000298023224 * Purity/3FRt3SRqqt*FBt/FRt_purity‚0.125 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.4551282823085785 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 :"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.43122613430023193 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 ,"Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.375 7 Purity/1FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.3710870146751404 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.2801888883113861 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 ;"Purity/3FRt*SRqqt1FBt/event_purity‚0.008333333767950535 7 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.2666666805744171 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.4658067524433136 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.5388957262039185 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 particle/accuracy_1_of_2‚1 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.4014025032520294 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.2619445025920868 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9974388480186462 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.5026707053184509 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_5_of_5‚1 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.6482142806053162 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 +jet/accuracy_3_of_6‚0.11538461595773697 particle/accuracy_6_of_6‚1 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 particle/accuracy_2_of_7‚1 9"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.3888888955116272 -"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.6875 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.3119843006134033 9"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.4193868637084961 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.08696950972080231 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 jet/accuracy_1_of_6‚1 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.2757250666618347 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.34739822149276733 6 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.992598831653595 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 jet/accuracy_7_of_7‚0 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 ) Purity/1FRt1SRqqt3FBt/FRt_purity‚0.25 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.4594617486000061 :"Purity/0FRt*SRqqt0FBt/event_purity‚0.35569268465042114 7 Purity/0FRt2SRqqt1FBt/FBt_purity‚0.9811320900917053 6 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.953703761100769 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.3061496317386627 *jet/accuracy_1_of_5‚0.9627304673194885 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5246641039848328 :"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.44352278113365173 7 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.4356060326099396 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 jet/accuracy_6_of_6‚0 jet/accuracy_2_of_7‚0.5 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.8888888955116272 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.015856919810175896 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 +jet/accuracy_1_of_1‚0.46646544337272644 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 particle/accuracy_1_of_7‚1 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.21266044676303864 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.39217233657836914 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.14327484369277954 7 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.2013511061668396 & Purity/1FRt*SRqqt4FBt/FRt_purity‚0 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5610036253929138 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 ("Purity/2FRt2SRqqt2FBt/SRqqt_purity‚1 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 particle/accuracy_4_of_7‚1 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.6105524301528931 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 :"Purity/0FRt0SRqqt2FBt/event_purity‚0.30163946747779846 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9923497438430786 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.31235629320144653 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 particle/accuracy_3_of_7‚1 ) Purity/4FRt1SRqqt*FBt/FRt_purity‚0.25 6 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.546884298324585 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9969857335090637 *jet/accuracy_1_of_4‚0.9064232707023621 7 Purity/3FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.4852508008480072 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9931841492652893 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.09626070410013199 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.4727446436882019 * Purity/*FRt*SRqqt4FBt/FBt_purity‚0.875 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 :"Purity/0FRt3SRqqt*FBt/event_purity‚0.03104575164616108 :"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.41470587253570557 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.28351643681526184 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9981811046600342 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.4895102381706238 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 jet/accuracy_5_of_6‚0 6 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.999028742313385 8 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.39118167757987976 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 :"Purity/0FRt2SRqqt1FBt/event_purity‚0.06509433686733246 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 jet/accuracy_4_of_7‚0 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.1388888955116272 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.7026745080947876 7"Purity/*FRt0SRqqt3FBt/event_purity‚0.15079365670681 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9952331185340881 :"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.35465461015701294 9"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.3681817650794983 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.24668805301189423 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9920634627342224 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.37142854928970337 *"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.5 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.06362208724021912 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 :"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.47067901492118835 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5733566880226135 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.5550040602684021 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 & Purity/0FRt3SRqqt1FBt/FBt_purity‚1 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 ."Purity/1FRt0SRqqt3FBt/event_purity‚0.15625 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.5174418687820435 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.21040602028369904 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.18168596923351288 8"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.465707927942276 8"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.579365074634552 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 & Purity/1FRt3SRqqt2FBt/FRt_purity‚1 ) Purity/4FRt1SRqqt0FBt/FRt_purity‚0.25 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.5919463634490967 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 particle/accuracy_1_of_6‚1 :"Purity/*FRt2SRqqt2FBt/event_purity‚0.03999999910593033 *"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.5 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 6validation_average_jet_accuracy‚0.4314018189907074 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.4562223553657532 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.24196341633796692 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9852729439735413 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9926124215126038 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 6 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.990164577960968 <"Purity/2FRt1SRqqt*FBt/event_purity‚0.0011337868636474013 +jet/accuracy_3_of_4‚0.09362613409757614 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 7 Purity/3FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 :"Purity/0FRt*SRqqt*FBt/event_purity‚0.45135942101478577 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5975102186203003 9"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.4209336042404175 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 ,jet/accuracy_3_of_3‚0.030177118256688118 particle/accuracy_3_of_3‚1 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.5178571343421936 :"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.47589877247810364 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.4942838251590729 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.4000000059604645 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.4031587243080139 :"Purity/*FRt3SRqqt*FBt/event_purity‚0.00994623638689518 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9937278032302856 :"Purity/2FRt0SRqqt2FBt/event_purity‚0.01315789483487606 <"Purity/2FRt1SRqqt0FBt/event_purity‚0.0015873016091063619 ."Purity/3FRt0SRqqt1FBt/event_purity‚0.03125 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 particle/accuracy_5_of_7‚1 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9871096014976501 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.2916666567325592 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5468379259109497 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 7 Purity/1FRt0SRqqt1FBt/FBt_purity‚0.9990079402923584 8"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.636805534362793 7 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.1866752803325653 :"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.46389445662498474 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 particle/accuracy_2_of_2‚1 particle/accuracy_2_of_5‚1 9"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.3888888955116272 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.1255529373884201 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6165039539337158 :"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.49511685967445374 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2867063283920288 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 + Purity/1FRt3SRqqt1FBt/FRt_purity‚0.3125 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.3571428656578064 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.4834769070148468 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.23341259360313416 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 7 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.9833333492279053 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 7 Purity/1FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 -"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.6875 :"Purity/*FRt0SRqqt2FBt/event_purity‚0.23000168800354004 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.3611111044883728 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 :"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.46485626697540283 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 <"Purity/1FRt2SRqqt*FBt/event_purity‚0.0010582010727375746 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.45186781883239746 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 9"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.4248366355895996 ("Purity/4FRt*SRqqt0FBt/SRqqt_purity‚1 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 & Purity/0FRt2SRqqt2FBt/FBt_purity‚1 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.9166666865348816 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚1 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.2283039093017578 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9990079402923584 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.5833333134651184 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 particle/accuracy_7_of_7‚1 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 * Purity/0FRt0SRqqt4FBt/FBt_purity‚0.875 :"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.47682473063468933 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 (particle/f_score‚0.42040422558784485 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 +"Purity/1FRt*SRqqt3FBt/SRqqt_purity‚0.75 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.22002968192100525 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.19966836273670197 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9912949204444885 particle/accuracy_2_of_4‚1 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.21266044676303864 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5930044054985046 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.21760809421539307 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.7575758099555969 +jet/accuracy_3_of_5‚0.11722990870475769 jet/accuracy_4_of_6‚0 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.016906864941120148 7 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.1681034415960312 6 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.228073850274086 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 +"Purity/0FRt*SRqqt4FBt/event_purity‚0.25 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.31459280848503113 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.20105890929698944 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 +"Purity/0FRt0SRqqt4FBt/event_purity‚0.25 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 :"Purity/0FRt1SRqqt0FBt/event_purity‚0.43924060463905334 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.4513021409511566 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 & Purity/*FRt3SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.017233561724424362 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 _step‚494 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9925049543380737 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.5603070259094238 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.20498083531856537 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.8571428656578064 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9865591526031494 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9980158805847168 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.17715394496917725 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.07028128206729889 ;"Purity/2FRt*SRqqt2FBt/event_purity‚0.011627906933426857 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9375000596046448 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.4390276372432709 , Purity/*FRt2SRqqt2FBt/FRt_purity‚0.28125 ) Purity/3FRt1SRqqt2FBt/FRt_purity‚0.25 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 *"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.5 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 7 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.2154836654663086 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.4340278208255768 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.4722139835357666 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 particle/accuracy_1_of_3‚1 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.1805555671453476 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.4905274212360382 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9969576597213745 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.5438596606254578 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.1771824061870575 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.18627983331680298 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 8 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.28425928950309753 & Purity/0FRt3SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.8999999761581421 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 :"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.49568966031074524 ,particle/sensitivity‚0.33426129817962646 trainer/global_step‚6044 % _timestamp‚1.7885174080546775e+09 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.04191208630800247 9"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.4518601596355438 /"Purity/0FRt0SRqqt3FBt/event_purity‚0.140625 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 jet/accuracy_3_of_7‚0 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9930326342582703 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.36319446563720703 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 7 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.1428571492433548 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9936468601226807 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.4871031939983368 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 ,jet/accuracy_4_of_5‚0.002016128972172737 +particle/specificity‚0.9527735114097595 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9907523989677429 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 (particle/accuracy‚0.8561972379684448 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.20000000298023224 * Purity/3FRt3SRqqt0FBt/FRt_purity‚0.125 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6534271836280823 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9841269850730896 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 jet/accuracy_5_of_5‚0 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.5789473652839661 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.25079578161239624 8 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.40528276562690735 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 7 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.5882353186607361 ;"Purity/*FRt1SRqqt2FBt/event_purity‚0.032629288733005524 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9919765591621399 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt2SRqqt2FBt/event_purity‚0.0714285746216774 6 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.339313805103302 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.4444444477558136 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 *jet/accuracy_2_of_4‚0.4061182737350464 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.3958333134651184 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.1691529005765915 :"Purity/*FRt0SRqqt4FBt/event_purity‚0.20000000298023224 particle/accuracy_3_of_6‚cÑf 1 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 ;"Purity/1FRt1SRqqt2FBt/event_purity‚0.010416666977107525 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 9"Purity/*FRt*SRqqt4FBt/event_purity‚0.1666666716337204 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.6666666865348816 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.039720334112644196 particle/accuracy_1_of_4‚1 particle/accuracy_2_of_6‚1 ) Purity/2FRt3SRqqt1FBt/FRt_purity‚0.25 * Purity/4FRt*SRqqt*FBt/FRt_purity‚0.125 ("Purity/*FRt2SRqqt3FBt/SRqqt_purity‚1 *jet/accuracy_1_of_3‚0.8750855922698975 *"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.5 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 7 Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9583333134651184 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 *jet/accuracy_1_of_2‚0.7530346512794495 particle/accuracy_4_of_4‚1 particle/accuracy_1_of_5‚1 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 :"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.41333332657814026 *loss/val_total_loss‚39.933971592619194 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.11980847269296646 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.02230686880648136 ,jet/accuracy_4_of_4‚0.008102045394480228 9"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.5166666507720947 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 ("Purity/4FRt*SRqqt*FBt/SRqqt_purity‚1 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9958333373069763 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 7 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.3259009122848511 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 particle/accuracy_4_of_6‚1 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.21463355422019958 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.3237974941730499 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.30138346552848816¦U›Œv醨 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.21266044676303864 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.39217233657836914 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.14327484369277954 7 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.2013511061668396 & Purity/1FRt*SRqqt4FBt/FRt_purity‚0 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5610036253929138 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 ("Purity/2FRt2SRqqt2FBt/SRqqt_purity‚1 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 particle/accuracy_4_of_7‚1 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.6105524301528931 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 :"Purity/0FRt0SRqqt2FBt/event_purity‚0.30163946747779846 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9923497438430786 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.31235629320144653 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 particle/accuracy_3_of_7‚1 ) Purity/4FRt1SRqqt*FBt/FRt_purity‚0.25 6 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.546884298324585 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9969857335090637 *jet/accuracy_1_of_4‚0.9064232707023621 7 Purity/3FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.4852508008480072 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9931841492652893 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.09626070410013199 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.4727446436882019 * Purity/*FRt*SRqqt4FBt/FBt_purity‚0.875 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 :"Purity/0FRt3SRqqt*FBt/event_purity‚0.03104575164616108 :"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.41470587253570557 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.28351643681526184 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9981811046600342 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.4895102381706238 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 jet/accuracy_5_of_6‚0 6 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.999028742313385 8 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.39118167757987976 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 :"Purity/0FRt2SRqqt1FBt/event_purity‚0.06509433686733246 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 jet/accuracy_4_of_7‚0 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.1388888955116272 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.7026745080947876 7"Purity/*FRt0SRqqt3FBt/event_purity‚0.15079365670681 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9952331185340881 :"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.35465461015701294 9"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.3681817650794983 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.24668805301189423 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9920634627342224 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.37142854928970337 *"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.5 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.06362208724021912 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 :"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.47067901492118835 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5733566880226135 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.5550040602684021 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 & Purity/0FRt3SRqqt1FBt/FBt_purity‚1 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 ."Purity/1FRt0SRqqt3FBt/event_purity‚0.15625 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.5174418687820435 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.21040602028369904 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.18168596923351288 8"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.465707927942276 8"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.579365074634552 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 & Purity/1FRt3SRqqt2FBt/FRt_purity‚1 ) Purity/4FRt1SRqqt0FBt/FRt_purity‚0.25 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.5919463634490967 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 particle/accuracy_1_of_6‚1 :"Purity/*FRt2SRqqt2FBt/event_purity‚0.03999999910593033 *"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.5 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 6validation_average_jet_accuracy‚0.4314018189907074 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.4562223553657532 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.24196341633796692 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9852729439735413 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9926124215126038 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 6 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.990164577960968 <"Purity/2FRt1SRqqt*FBt/event_purity‚0.0011337868636474013 +jet/accuracy_3_of_4‚0.09362613409757614 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 7 Purity/3FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 :"Purity/0FRt*SRqqt*FBt/event_purity‚0.45135942101478577 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5975102186203003 9"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.4209336042404175 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 ,jet/accuracy_3_of_3‚0.030177118256688118 particle/accuracy_3_of_3‚1 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.5178571343421936 :"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.47589877247810364 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.4942838251590729 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.4000000059604645 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.4031587243080139 :"Purity/*FRt3SRqqt*FBt/event_purity‚0.00994623638689518 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9937278032302856 :"Purity/2FRt0SRqqt2FBt/event_purity‚0.01315789483487606 <"Purity/2FRt1SRqqt0FBt/event_purity‚0.0015873016091063619 ."Purity/3FRt0SRqqt1FBt/event_purity‚0.03125 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 particle/accuracy_5_of_7‚1 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9871096014976501 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.2916666567325592 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5468379259109497 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 7 Purity/1FRt0SRqqt1FBt/FBt_purity‚0.9990079402923584 8"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.636805534362793 7 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.1866752803325653 :"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.46389445662498474 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 particle/accuracy_2_of_2‚1 particle/accuracy_2_of_5‚1 9"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.3888888955116272 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.1255529373884201 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6165039539337158 :"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.49511685967445374 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2867063283920288 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 + Purity/1FRt3SRqqt1FBt/FRt_purity‚0.3125 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.3571428656578064 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.4834769070148468 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.23341259360313416 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 7 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.9833333492279053 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 7 Purity/1FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 -"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.6875 :"Purity/*FRt0SRqqt2FBt/event_purity‚0.23000168800354004 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.3611111044883728 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 :"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.46485626697540283 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 <"Purity/1FRt2SRqqt*FBt/event_purity‚0.0010582010727375746 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.45186781883239746 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 9"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.4248366355895996 ("Purity/4FRt*SRqqt0FBt/SRqqt_purity‚1 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 & Purity/0FRt2SRqqt2FBt/FBt_purity‚1 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.9166666865348816 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚1 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.2283039093017578 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9990079402923584 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.5833333134651184 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 particle/accuracy_7_of_7‚1 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 * Purity/0FRt0SRqqt4FBt/FBt_purity‚0.875 :"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.47682473063468933 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 (particle/f_score‚0.42040422558784485 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 +"Purity/1FRt*SRqqt3FBt/SRqqt_purity‚0.75 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.22002968192100525 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.19966836273670197 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9912949204444885 particle/accuracy_2_of_4‚1 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.21266044676303864 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5930044054985046 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.21760809421539307 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.7575758099555969 +jet/accuracy_3_of_5‚0.11722990870475769 jet/accuracy_4_of_6‚0 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.016906864941120148 7 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.1681034415960312 6 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.228073850274086 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 +"Purity/0FRt*SRqqt4FBt/event_purity‚0.25 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.31459280848503113 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.20105890929698944 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 +"Purity/0FRt0SRqqt4FBt/event_purity‚0.25 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 :"Purity/0FRt1SRqqt0FBt/event_purity‚0.43924060463905334 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.4513021409511566 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 & Purity/*FRt3SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.017233561724424362 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 _step‚494 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9925049543380737 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.5603070259094238 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.20498083531856537 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.8571428656578064 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9865591526031494 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9980158805847168 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.17715394496917725 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.07028128206729889 ;"Purity/2FRt*SRqqt2FBt/event_purity‚0.011627906933426857 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9375000596046448 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.4390276372432709 , Purity/*FRt2SRqqt2FBt/FRt_purity‚0.28125 ) Purity/3FRt1SRqqt2FBt/FRt_purity‚0.25 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 *"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.5 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 7 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.2154836654663086 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.4340278208255768 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.4722139835357666 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 particle/accuracy_1_of_3‚1 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.1805555671453476 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.4905274212360382 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9969576597213745 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.5438596606254578 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.1771824061870575 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.18627983331680298 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 8 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.28425928950309753 & Purity/0FRt3SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.8999999761581421 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 :"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.49568966031074524 ,particle/sensitivity‚0.33426129817962646 trainer/global_step‚6044 % _timestamp‚1.7885174080546775e+09 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.04191208630800247 9"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.4518601596355438 /"Purity/0FRt0SRqqt3FBt/event_purity‚0.140625 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 jet/accuracy_3_of_7‚0 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9930326342582703 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.36319446563720703 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 7 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.1428571492433548 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9936468601226807 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.4871031939983368 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 ,jet/accuracy_4_of_5‚0.002016128972172737 +particle/specificity‚0.9527735114097595 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9907523989677429 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 (particle/accuracy‚0.8561972379684448 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.20000000298023224 * Purity/3FRt3SRqqt0FBt/FRt_purity‚0.125 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6534271836280823 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9841269850730896 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 jet/accuracy_5_of_5‚0 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.5789473652839661 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.25079578161239624 8 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.40528276562690735 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 7 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.5882353186607361 ;"Purity/*FRt1SRqqt2FBt/event_purity‚0.032629288733005524 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9919765591621399 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt2SRqqt2FBt/event_purity‚0.0714285746216774 6 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.339313805103302 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.4444444477558136 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 *jet/accuracy_2_of_4‚0.4061182737350464 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.3958333134651184 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.1691529005765915 :"Purity/*FRt0SRqqt4FBt/event_purity‚0.20000000298023224 particle/accuracy_3_of_6‚1 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 ;"Purity/1FRt1SRqqt2FBt/event_purity‚0.010416666977107525 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 9"Purity/*FRt*SRqqt4FBt/event_purity‚0.1666666716337204 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.6666666865348816 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.039720334112644196 particle/accuracy_1_of_4‚1 particle/accuracy_2_of_6‚1 ) Purity/2FRt3SRqqt1FBt/FRt_purity‚0.25 * Purity/4FRt*SRqqt*FBt/FRt_purity‚0.125 ("Purity/*FRt2SRqqt3FBt/SRqqt_purity‚1 *jet/accuracy_1_of_3‚0.8750855922698975 *"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.5 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 7 Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9583333134651184 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 *jet/accuracy_1_of_2‚0.7530346512794495 particle/accuracy_4_of_4‚1 particle/accuracy_1_of_5‚1 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 :"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.41333332657814026 *loss/val_total_loss‚39.933971592619194 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.11980847269296646 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.02230686880648136 ,jet/accuracy_4_of_4‚0.008102045394480228 9"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.5166666507720947 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 ("Purity/4FRt*SRqqt*FBt/SRqqt_purity‚1 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9958333373069763 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 7 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.3259009122848511 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 particle/accuracy_4_of_6‚1 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.21463355422019958 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.3237974941730499 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.30138346552848816 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 ;"Purity/3FRt0SRqqt*FBt/event_purity‚0.011111111380159855 particle/accuracy_6_of_7‚1 ) Purity/*FRt1SRqqt3FBt/FRt_purity‚0.25 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.5247529149055481 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 +jet/accuracy_2_of_2‚0.11170494556427002 & Purity/1FRt0SRqqt4FBt/FRt_purity‚0 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 7 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.2196120172739029 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9766082167625427 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.4692222774028778 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.10089284926652908 particle/accuracy_1_of_1‚1 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.28549379110336304 9"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.4000000059604645 :"Purity/0FRt3SRqqt0FBt/event_purity‚0.03104575164616108 ,"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.875 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 <"Purity/1FRt2SRqqt0FBt/event_purity‚0.0012210012646391988 9"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.4309942424297333 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 *jet/accuracy_2_of_3‚0.2587764859199524 particle/accuracy_2_of_3‚1 particle/accuracy_3_of_5‚1 :"Purity/*FRt3SRqqt0FBt/event_purity‚0.00994623638689518 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 particle/accuracy_5_of_6‚1 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.9074073433876038 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9931722283363342 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 _runtime‚7033.527134075 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.4059998393058777 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.22158724069595337 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.4555996358394623 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.4270833134651184 *"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.5 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.7142857313156128 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.027457214891910553 :"Purity/0FRt*SRqqt2FBt/event_purity‚0.25389009714126587 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.59375 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.1130499392747879 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.010366199538111687 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 * Purity/4FRt*SRqqt0FBt/FRt_purity‚0.125 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚1 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 -"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.4375 jet/accuracy_6_of_7‚0 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 & Purity/*FRt3SRqqt2FBt/FRt_purity‚1 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.4232521057128906 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 & Purity/*FRt0SRqqt4FBt/FRt_purity‚0 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.33762624859809875 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 & Purity/*FRt*SRqqt4FBt/FRt_purity‚0 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.11347518116235733 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 & Purity/*FRt3SRqqt1FBt/FBt_purity‚1 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.3333333432674408 9"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.2916666567325592 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.04435495287179947 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 *jet/accuracy_2_of_5‚0.4909690022468567 ("Purity/0FRt2SRqqt3FBt/SRqqt_purity‚1 epoch‚4 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9290780425071716 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.22666670382022858 jet/accuracy_5_of_7‚0 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9404761791229248 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.23269513249397278 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.19976213574409485 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 particle/accuracy_3_of_4‚1 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.3611111044883728 9"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.4000000059604645 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.2222222238779068 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 7 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.8571428656578064 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 6 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.274755597114563 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.22949247062206268 8"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.521170437335968 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 <"Purity/3FRt*SRqqt*FBt/event_purity‚0.0023419205099344254 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9900000095367432 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.6040292382240295 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.9122807383537292 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8080624938011169 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.5211611390113831 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.20164605975151062 ,&Purity/4FRt3SRqqt*FBä²Êît/event_proportion‚0 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 +"Purity/1FRt1SRqqt3FBt/SRqqt_purity‚0.75 :"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.47789621353149414 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.1805555671453476 *"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.5 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.26102006435394287 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 :"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.43924060463905334 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 7 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.3083333373069763 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9938737750053406 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 jet/accuracy_1_of_7‚1 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9990079402923584 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5272077322006226 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 8 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.35462963581085205 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.19696968793869019 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.5170079469680786 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.5196301937103271 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.19421087205410004 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.7575758099555969 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.10526315867900848 +jet/accuracy_2_of_6‚0.49358975887298584 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.23135557770729065 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.02986210398375988 ;"Purity/0FRt1SRqqt2FBt/event_purity‚0.056752875447273254 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.02551707811653614 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_4_of_5‚1 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 6 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.998742938041687 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9826388955116272 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.20000000298023224 * Purity/3FRt3SRqqt*FBt/FRt_purity‚0.125 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.4551282823085785 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 :"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.43122613430023193 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 ,"Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.375 7 Purity/1FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.3710870146751404 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.2801888883113861 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 ;"Purity/3FRt*SRqqt1FBt/event_purity‚0.008333333767950535 7 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.2666666805744171 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.4658067524433136 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.5388957262039185 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 particle/accuracy_1_of_2‚1 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.4014025032520294 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.2619445025920868 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9974388480186462 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.5026707053184509 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_5_of_5‚1 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.6482142806053162 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 +jet/accuracy_3_of_6‚0.11538461595773697 particle/accuracy_6_of_6‚1 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 particle/accuracy_2_of_7‚1 9"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.3888888955116272 -"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.6875 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.3119843006134033 9"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.4193868637084961 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.08696950972080231 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 jet/accuracy_1_of_6‚1 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.2757250666618347 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.34739822149276733 6 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.992598831653595 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 jet/accuracy_7_of_7‚0 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 ) Purity/1FRt1SRqqt3FBt/FRt_purity‚0.25 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.4594617486000061 :"Purity/0FRt*SRqqt0FBt/event_purity‚0.35569268465042114 7 Purity/0FRt2SRqqt1FBt/FBt_purity‚0.9811320900917053 6 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.953703761100769 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.3061496317386627 *jet/accuracy_1_of_5‚0.9627304673194885 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5246641039848328 :"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.44352278113365173 7 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.4356060326099396 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 jet/accuracy_6_of_6‚0 jet/accuracy_2_of_7‚0.5 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.8888888955116272 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.015856919810175896 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 +jet/accuracy_1_of_1‚0.46646544337272644 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 particle/accuracy_1_of_7‚1îg½Iì`ê‚(:Õ ¡¸êÔÞЄ€ disk.sdb2.in‚ 0.92578125 disk.sda1.out‚0 disk.nvme0n1.out‚ 36796.4375 cpu‚25.226414701681556 network.recv‚ 2.247754e+06 disk.rbd0.out‚ 1.68359375 disk.sdb1.in‚0 disk.md0.out‚778.3125 disk.rbd2.in‚0 disk.sdb2.out‚807.7431640625 disk.sdb.in‚ 0.92578125 network.sent‚ 1.5210971e+07) disk./.usagePercent‚75.85135156813168 disk.rbd0.in‚0# disk.nvme1n1.out‚44606.62109375 disk.dm-0.out‚81403.05859375 proc.cpu.threads‚17 disk.nvme1n1.in‚ 498.80078125 disk.md0p1.in‚5.3125 disk.sdb.out‚807.7431640625$ memory_percent‚93.93668174743652 disk.sda2.out‚807.7431640625 disk.md0.in‚5.3125 disk.rbd2.out‚ 1.845703125 disk.sda.out‚807.7431640625 disk.nvme0n1.in‚ 603.58984375 disk.sdb1.out‚0 disk.sda1.in‚0 disk.sda.in‚ 4.38671875* proc.memory.percent‚20.201539993286133 disk.dm-0.in‚ 1102.390625 disk.md0p1.out‚778.3125' proc.memory.availableMB‚ 993.4140625" proc.memory.rssMB‚ 3309.8203125$ disk./.usageGB‚2712.811737060547 disk.sda2.in‚ 4.38671875–Ù¥^Žë‚(:ƒ ¡¸êԤ߀ gpu.0.gpu‚7.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚33.0 gpu.0.powerWatts‚85.56' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚21.39 gpu.0.smClock‚1155 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 P2ì- _wandbruntime‚7043  _runtime‚7043QK1Åí¿ % lr-AdamW/pg1‚0.000502831572700479 % lr-AdamW/pg2‚0.000502831572700479 trainer/global_step‚6049 % _timestamp‚1.7885174179332454e+09 _runtime‚7043.403580381 _step‚495Ž¼ÒžÊîÄ % lr-AdamW/pg2‚0.000502831572700479 trainer/global_step‚6049 % _timestamp‚1.7885174179332454e+09 _runtime‚7043.403580381 _step‚495 % lr-AdamW/pg1‚0.000502831572700479ï¶e2ï- _wandbruntime‚7043  _runtime‚7043Z•·Zù'ðóO XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚77854455 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 _runtime‚7043.404749606 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177047040 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚505551472029184 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 151944350 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6888727636992 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1187 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 125205494 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1212 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚498663643800064 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚565060685938176 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 125205496 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚36446606 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 151970438 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚547 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 398533632 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚505551472029184 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 151970438 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚557868168398848 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _step‚496 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚498663661803008 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚36447153 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214665 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚77854453 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 220471296 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6887828229120 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚549 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7192273127936 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407872 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6862154538348 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 151944350 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚498663661803008 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450780 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚505552389440000 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410816 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6888727636992 trainer/global_step‚6049 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 114301608 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚505552389440000 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411420 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 " _timestamp‚1.7885174179343e+09 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 125205496 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955591168 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚565060441526784 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚487478724893840 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26088 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6863051949768 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7192493599232 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6887828229120 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚557868192338944 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917410816 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26088 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 114301059 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚494340862392828 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26090 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0’vÙÉþ'ñøO VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007241 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 125205494 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955591168 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚498663643800064 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 151970438 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚547 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 398533632 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚505551472029184 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 151970438 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚557868168398848 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _step‚496 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚498663661803008 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚36447153 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214665 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚77854453 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 220471296 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6887828229120 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚549 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7192273127936 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407872 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6862154538348 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 151944350 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚498663661803008 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450780 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚505552389440000 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410816 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6888727636992 trainer/global_step‚6049 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 114301608 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚505552389440000 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411420 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 " _timestamp‚1.7885174179343e+09 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 125205496 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955591168 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚565060441526784 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚487478724893840 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26088 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6863051949768 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7192493599232 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6887828229120 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚557868192338944 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917410816 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26088 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 114301059 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚494340862392828 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26090 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚77854455 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 _runtime‚7043.404749606 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177047040 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚505551472029184 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 151944350 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6888727636992 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1187 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 125205494 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1212 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚498663643800064 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚565060685938176 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 125205496 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚36446606 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 JDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 277220449 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26114 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007241 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 151968941 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚6049 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6888941032448 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6863263199666 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 151995055 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 125225392 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6864160611294 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 114319573 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚36452742 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 _step‚497 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899421184 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚494419951148274 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26114 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚498743274095616 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚487556687948608 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 125225392 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚505633132552192 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6889840453632 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 151995055 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚494420865599262 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚498743292098560 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6889840453632 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 % _timestamp‚1.7885174188865144e+09 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚498743274095616 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚565150921384960 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1187 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 151968941 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 114319013 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177047040 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚560 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚36452184 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚77866829 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚557957487619584 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7193433765376 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 246495232c1¡ 'ô…N L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 114319573 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚36452742 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 _step‚497 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899421184 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚494419951148274 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26114 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚498743274095616 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚487556687948608 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 125225392 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚505633132552192 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6889840453632 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 151995055 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚494420865599262 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚498743292098560 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6889840453632 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 % _timestamp‚1.7885174188865144e+09 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚498743274095616 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚565150921384960 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1187 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 151968941 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 114319013 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177047040 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚560 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚36452184 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚77866829 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚557957487619584 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7193433765376 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 246495232 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31686 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚557957511559680 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450988 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 277194333 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7193656320512 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177047040 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚498743292098560 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6888941032448 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 398533632 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 277220449 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26114 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007241 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 151968941 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚6049 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6888941032448 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6863263199666 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 151995055 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 125225392 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6864160611294ñ‚¿)™2õ- _wandbruntime‚7044  _runtime‚7044xô9w^öØ 1loss/SRqqt4/detection_loss‚2.4443619772791862 /loss/FRt2/assignment_loss‚1.661477072238922 /loss/FRt2/detection_loss‚1.7484424449130893 /loss/FRt3/detection_loss‚2.0969574670899997 1loss/SRqqt3/detection_loss‚1.3356935821237073 .loss/FRt1/detection_loss‚2.175126515543878 2loss/SRqqt4/assignment_loss‚1.3654686497258288 1loss/SRqqt1/detection_loss‚1.5750925324570675 /loss/FBt2/detection_loss‚0.6154296317700904 _step‚498 2loss/SRqqt1/assignment_loss‚1.3932474488034272 &loss/total_loss‚39.632861048625266 epoch‚5 _runtime‚7044.358824059 2loss/SRqqt2/assignment_loss‚0.9462827349761649 1loss/FBt1/assignment_loss‚0.11495409731278983 /loss/FBt3/detection_loss‚2.4189795553684235 .loss/FBt4/detection_loss‚2.877880887551741 % _timestamp‚1.7885174188886955e+09 2loss/SRqqt3/assignment_loss‚1.2685833453304238 1loss/FBt2/assignment_loss‚0.15361544775983338 0loss/FBt3/assignment_loss‚0.6833942669031785 /loss/FBt1/detection_loss‚2.9631600379943848 trainer/global_step‚6049 /loss/FRt1/assignment_loss‚2.502959526720501 /loss/FRt3/assignment_loss‚2.499395915594968 .loss/FRt4/assignment_loss‚3.02280917391181 3loss/FBt4/assignment_loss‚0.0006317178212239841 /loss/FRt4/detection_loss‚2.8362590074539185 1loss/SRqqt2/detection_loss‚0.9326580119807095Ó-'´c÷Ý epoch‚5 _runtime‚7044.358824059 2loss/SRqqt2/assignment_loss‚0.9462827349761649 1loss/FBt1/assignment_loss‚0.11495409731278983 /loss/FBt3/detection_loss‚2.4189795553684235 .loss/FBt4/detection_loss‚2.877880887551741 % _timestamp‚1.7885174188886955e+09 2loss/SRqqt3/assignment_loss‚1.2685833453304238 1loss/FBt2/assignment_loss‚0.15361544775983338 0loss/FBt3/assignment_loss‚0.6833942669031785 /loss/FBt1/detection_loss‚2.9631600379943848 trainer/global_step‚6049 /loss/FRt1/assignment_loss‚2.502959526720501 /loss/FRt3/assignment_loss‚2.499395915594968 .loss/FRt4/assignment_loss‚3.02280917391181 3loss/FBt4/assignment_loss‚0.0006317178212239841 /loss/FRt4/detection_loss‚2.8362590074539185 1loss/SRqqt2/detection_loss‚0.9326580119807095 1loss/SRqqt4/detection_loss‚2.4443619772791862 /loss/FRt2/assignment_loss‚1.661477072238922 /loss/FRt2/detection_loss‚1.7484424449130893 /loss/FRt3/detection_loss‚2.0969574670899997 1loss/SRqqt3/detection_loss‚1.3356935821237073 .loss/FRt1/detection_loss‚2.175126515543878 2loss/SRqqt4/assignment_loss‚1.3654686497258288 1loss/SRqqt1/detection_loss‚1.5750925324570675 /loss/FBt2/detection_loss‚0.6154296317700904 _step‚498 2loss/SRqqt1/assignment_loss‚1.3932474488034272 &loss/total_loss‚39.632861048625266òD Š”ø‚(:‰ °¸êÔÎÓÇþ gpu.0.gpu‚54.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚169.333' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.33325 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚00e:¾kù‚(:à °¸êÔœ²ûþ disk.sda.in‚ 4.38671875! disk.sda2.out‚809.63623046875 disk.rbd0.in‚0 disk.rbd0.out‚ 1.76171875 disk.dm-0.in‚ 1102.390625) disk./.usagePercent‚75.85132831613566# disk.nvme0n1.out‚38269.66015625 disk.sdb1.out‚0 network.sent‚ 1.5321836e+07! disk.sdb2.out‚809.63623046875 disk.dm-0.out‚82876.48046875# memory_percent‚88.0873441696167) proc.memory.availableMB‚ 1951.76953125* proc.memory.percent‚18.320131301879883 cpu‚25.22424669411213 disk.md0.out‚ 780.140625 disk.rbd2.out‚ 1.857421875 disk.sdb1.in‚0" proc.memory.rssMB‚ 3001.5703125 disk.sdb.out‚809.63623046875 network.recv‚ 2.252995e+06 disk.rbd2.in‚0 disk.sdb2.in‚ 0.92578125 disk.sda1.out‚0 disk.sdb.in‚ 0.92578125$ disk./.usageGB‚2712.810905456543 disk.md0.in‚5.3125 disk.md0p1.out‚ 780.140625 disk.sda1.in‚0 disk.sda.out‚809.63623046875 disk.sda2.in‚ 4.38671875" disk.nvme1n1.out‚ 44606.8203125 disk.md0p1.in‚5.3125 disk.nvme0n1.in‚ 603.58984375 disk.nvme1n1.in‚ 498.80078125 proc.cpu.threads‚17’ÞÙØ”ú‚(:‰ ¿¸êÔʨ¦ÿ gpu.0.gpu‚54.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚170.047' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.51175 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0fg°mû‚(:â ¿¸êÔáí±€ disk.rbd2.in‚0" proc.memory.rssMB‚ 3001.5703125 disk.dm-0.out‚82949.07421875 disk.rbd0.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 cpu‚25.22383226896137 network.sent‚ 1.5429489e+07 disk.nvme0n1.in‚ 603.58984375 disk.sda.out‚811.15576171875# disk.nvme1n1.out‚44607.15234375 disk.sdb1.in‚0$ memory_percent‚88.08741569519043 network.recv‚ 2.258064e+06 disk.sda2.in‚ 4.38671875! disk.sda2.out‚811.15576171875 disk.sdb1.out‚0 disk.rbd2.out‚ 1.919921875( proc.memory.availableMB‚ 1951.7578125) disk./.usagePercent‚75.85133194259377 disk.dm-0.in‚ 1102.390625 proc.cpu.threads‚17 disk.md0p1.out‚ 781.59765625 disk.sda.in‚ 4.38671875 disk.md0.out‚ 781.59765625* proc.memory.percent‚18.320131301879883 disk.md0p1.in‚5.3125! disk.nvme0n1.out‚ 38341.921875 disk.sdb2.in‚ 0.92578125! disk.sdb2.out‚811.15576171875# disk./.usageGB‚2712.81103515625 disk.sdb.out‚811.15576171875 disk.sdb.in‚ 0.92578125 disk.rbd0.out‚ 1.76171875 disk.nvme1n1.in‚ 498.80078125 disk.md0.in‚5.3125@B*J”ü‚(:‰ θêÔ¦èÝ€ gpu.0.gpu‚41.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚39.0 gpu.0.powerWatts‚178.629' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚44.65725 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ßôIÀiý‚(:Þ θêÔ™Ñû€ disk.md0.in‚5.3125 disk.sda1.in‚0 disk.dm-0.in‚ 1102.390625! disk.sda2.out‚812.53955078125 network.recv‚ 2.262743e+06 disk.rbd0.out‚ 1.84912109375 disk.nvme0n1.in‚ 603.58984375$ memory_percent‚88.08646202087402 cpu‚25.22339067014436 network.sent‚ 1.5438039e+07$ disk./.usageGB‚2712.724452972412# disk.nvme1n1.out‚44607.46484375 disk.sdb1.out‚0 disk.md0p1.out‚782.9375( disk./.usagePercent‚75.8489110684962 disk.rbd0.in‚0 proc.cpu.threads‚17 disk.md0.out‚782.9375 disk.sdb2.in‚ 0.92578125" disk.nvme0n1.out‚ 38414.7734375" proc.memory.rssMB‚ 3001.5703125 disk.sdb.out‚812.53955078125 disk.sda.out‚812.53955078125 disk.nvme1n1.in‚ 498.80078125 disk.rbd2.in‚0 disk.rbd2.out‚ 1.919921875! disk.sdb2.out‚812.53955078125 disk.sda1.out‚0* proc.memory.percent‚18.320131301879883 disk.sdb.in‚ 0.92578125 disk.sda.in‚ 4.38671875 disk.sdb1.in‚0 disk.dm-0.out‚83022.23828125 disk.md0p1.in‚5.3125( proc.memory.availableMB‚ 1951.9140625 disk.sda2.in‚ 4.38671875l‰y×’þ‚(:‡ ݸêÔñ±¦ÿ gpu.0.gpu‚56.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚171.944' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.986 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0õöh{rÿ‚(:ç ݸêÔðú†€" proc.memory.rssMB‚ 3001.5703125 disk.rbd0.out‚ 1.84912109375 disk.sdb.in‚ 0.92578125# memory_percent‚88.0903959274292 network.sent‚ 1.5443334e+07 disk.md0.in‚5.3125 disk.rbd2.in‚0* proc.memory.percent‚18.320131301879883 disk.md0p1.out‚ 785.1953125 disk.nvme0n1.in‚ 603.58984375" disk.nvme0n1.out‚ 38558.8671875 disk.sda.out‚814.85791015625# disk.nvme1n1.out‚44607.70703125 disk.sda1.in‚0 disk.dm-0.in‚ 1102.390625! disk.sda2.out‚814.85791015625 cpu‚25.222996963588006 disk.rbd0.in‚0 disk.nvme1n1.in‚ 498.80078125 disk.sdb1.in‚0 disk.rbd2.out‚ 1.9990234375 disk.sdb2.in‚ 0.92578125 disk.sda1.out‚0 proc.cpu.threads‚17 network.recv‚ 2.266386e+06 disk.md0.out‚ 785.1953125! disk.sdb2.out‚814.85791015625 disk.sdb.out‚814.85791015625 disk.sda.in‚ 4.38671875 disk.sda2.in‚ 4.38671875 disk.sdb1.out‚0( disk./.usagePercent‚75.8508804485613% disk./.usageGB‚2712.7948875427246 disk.dm-0.out‚83166.57421875 disk.md0p1.in‚5.3125) proc.memory.availableMB‚ 1951.26953125ý(ñ 2€- _wandbruntime‚7095  _runtime‚7095ï–ÇÁ % _timestamp‚1.7885174696850007e+09 _runtime‚7095.155359599 _step‚499 & lr-AdamW/pg1‚0.0004967259498411866 & lr-AdamW/pg2‚0.0004967259498411866 trainer/global_step‚6099…$oWÌ‚Æ _runtime‚7095.155359599 _step‚499 & lr-AdamW/pg1‚0.0004967259498411866 & lr-AdamW/pg2‚0.0004967259498411866 trainer/global_step‚6099 % _timestamp‚1.7885174696850007e+09ó‰_hC2ƒ- _wandbruntime‚7095  _runtime‚7095Õ"d®„öO F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 279377646 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚509588641387008 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 248605696 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚498295313756336 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚569584213522432 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407872 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.–BÏNpeak‚35 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 _step‚500 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6943482809344 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1212 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚36727503 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚562334129435648 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7250084086784 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚78473255 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚617 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7250308752384 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 % _timestamp‚1.7885174696860197e+09 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177047040 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1212 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚36727503 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚562334129435648 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7250084086784 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚78473255 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚617 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7250308752384 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 % _timestamp‚1.7885174696860197e+09 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177047040 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚509588641387008 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 248605696 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚498295313756336 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚569584213522432 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407872 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 _step‚500 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6943482809344ôeÚcª2†- _wandbruntime‚7096  _runtime‚7096ûñ‚^'‡€N OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 153227957 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 153201843 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899421184 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26114 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 115218158 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 279448251 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚78485629 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚498375316962770 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450988 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1187 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955591168 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6944595612672 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚491455692655008 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31686 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 _runtime‚7096.136078396 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 126220292 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6945495033856 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚498374402511782 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 279422135 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 126220294 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚562423448656384 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007241 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 279448251 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚509670301910016 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 115218764 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚569674691514880 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 trainer/global_step‚6099 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 126220294 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚502724806876160 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7251242858496 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177047040 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7251469607936 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚78485631 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 398533632 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 126220292 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26114 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚604 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚562423472596480 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6919607268402 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚509670301910016 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177047040 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899421184 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚502724788873216 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 153201843 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚606É%ÎVˆ…N B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 126220292 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26114 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚604 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚562423472596480 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6919607268402 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚509670301910016 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177047040 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899421184 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚502724788873216 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 153201843 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚606 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 153227957 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 153201843 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899421184 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26114 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 115218158 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 279448251 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚78485629 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚498375316962770 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450988 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1187 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955591168 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6944595612672 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚491455692655008 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31686 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 _runtime‚7096.136078396 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 126220292 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6945495033856 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚498374402511782 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 279422135 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 126220294 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚562423448656384 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007241 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 279448251 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚509670301910016 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 115218764 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚569674691514880 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 trainer/global_step‚6099 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 126220294 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚502724806876160 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7251242858496 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177047040 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7251469607936 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚78485631 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 398533632õ€vî2‰- _wandbruntime‚7096  _runtime‚7096£[ŠÕ 0loss/FRt2/assignment_loss‚1.4968136795189069 1loss/SRqqt1/detection_loss‚1.3852644999112402 1loss/SRqqt2/detection_loss‚0.9871767297209728 _step‚502 -loss/FRt4/detection_loss‚2.96108520279328 1loss/SRqqt3/detection_loss‚2.5439728407517954 .loss/FRt2/detection_loss‚2.083718838656078 0loss/SRqqt4/detection_loss‚2.704706701967451 /loss/FBt1/detection_loss‚2.3203269243240356 .loss/FBt4/detection_loss‚2.347228470775816 2loss/SRqqt3/assignment_loss‚1.4302414290278274 1loss/FBt4/assignment_loss‚0.13899255093783816 /loss/FBt2/detection_loss‚0.6365794562193197 &loss/total_loss‚41.372154914650025 trainer/global_step‚6099 /loss/FRt1/detection_loss‚1.8181932380176835 .loss/FRt3/detection_loss‚2.318107984960079 _runtime‚ 7096.13711545 % _timestamp‚1.7885174706669476e+09 0loss/FRt1/assignment_loss‚2.4100484613512383 2loss/SRqqt2/assignment_loss‚1.0532282366757355 2loss/SRqqt4/assignment_loss‚1.4164419422547023 1loss/FBt2/assignment_loss‚0.16322379779073481 1loss/FBt3/assignment_loss‚0.43068081672595326 /loss/FBt3/detection_loss‚2.8133050948381424 /loss/FRt3/assignment_loss‚2.788031448258294 0loss/FRt4/assignment_loss‚3.3754285722970963 0loss/FBt1/assignment_loss‚0.2077002972312414 2loss/SRqqt1/assignment_loss‚1.5416576996445657 epoch‚5þj^Ç`‹Ú 0loss/FRt2/assignment_loss‚1.4968136795189069 1loss/SRqqt1/detection_loss‚1.3852644999112402 1loss/SRqqt2/detection_loss‚0.9871767297209728 _step‚502 -loss/FRt4/detection_loss‚2.96108520279328 1loss/SRqqt3/detection_loss‚2.5439728407517954 .loss/FRt2/detection_loss‚2.083718838656078 0loss/SRqqt4/detection_loss‚2.704706701967451 /loss/FBt1/detection_loss‚2.3203269243240356 .loss/FBt4/detection_loss‚2.347228470775816 2loss/SRqqt3/assignment_loss‚1.4302414290278274 1loss/FBt4/assignment_loss‚0.13899255093783816 /loss/FBt2/detection_loss‚0.6365794562193197 &loss/total_loss‚41.372154914650025 trainer/global_step‚6099 /loss/FRt1/detection_loss‚1.8181932380176835 .loss/FRt3/detection_loss‚2.318107984960079 _runtime‚ 7096.13711545 % _timestamp‚1.7885174706669476e+09 0loss/FRt1/assignment_loss‚2.4100484613512383 2loss/SRqqt2/assignment_loss‚1.0532282366757355 2loss/SRqqt4/assignment_loss‚1.4164419422547023 1loss/FBt2/assignment_loss‚0.16322379779073481 1loss/FBt3/assignment_loss‚0.43068081672595326 /loss/FBt3/detection_loss‚2.8133050948381424 /loss/FRt3/assignment_loss‚2.788031448258294 0loss/FRt4/assignment_loss‚3.3754285722970963 0loss/FBt1/assignment_loss‚0.2077002972312414 2loss/SRqqt1/assignment_loss‚1.5416576996445657 epoch‚5öáŽôÿ”Œ‚(:‰ ì¸êÔï÷Òþ gpu.0.gpu‚40.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚171.777' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.94425 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0›¹ÝRr‚(:ç ì¸êÔ‡ßíþ disk.sdb.out‚816.33056640625) proc.memory.availableMB‚ 1949.87109375 network.recv‚ 2.271096e+06% disk./.usageGB‚2712.8654022216797 disk.sda.out‚816.33056640625" disk.nvme1n1.out‚ 44607.9140625 disk.md0.in‚5.3125 disk.rbd2.in‚0$ memory_percent‚88.09893131256104 disk.sdb1.in‚0 disk.dm-0.out‚83166.84765625" proc.memory.rssMB‚ 3001.5703125) disk./.usagePercent‚75.85285206849757 disk.md0p1.in‚5.3125 disk.nvme0n1.in‚ 603.58984375 disk.sdb.in‚ 0.92578125 disk.nvme1n1.in‚ 498.80078125 disk.sdb1.out‚0 disk.md0.out‚ 786.6015625 disk.sdb2.in‚ 0.92578125 disk.sda1.in‚0 cpu‚25.222572103185904 disk.dm-0.in‚ 1102.390625# disk.nvme0n1.out‚38558.93359375 proc.cpu.threads‚17 disk.sda2.in‚ 4.38671875 disk.rbd0.out‚ 1.912109375 disk.rbd2.out‚ 1.9990234375 network.sent‚ 1.5447168e+07 disk.sda.in‚ 4.38671875* proc.memory.percent‚18.320131301879883! disk.sda2.out‚816.33056640625 disk.rbd0.in‚0! disk.sdb2.out‚816.33056640625 disk.sda1.out‚0 disk.md0p1.out‚ 786.6015625ëÄv©žŽ‚(:“ û¸êÔ»ñ¸ÿ gpu.0.gpu‚40.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚37.0 gpu.0.powerWatts‚118.474' gpu.0.enforcedPowerLimitWatts‚400.0) gpu.0.powerPercent‚29.618500000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0@¥às‚(:è û¸êÔÿ¬€ disk.rbd2.out‚ 2.06201171875$ memory_percent‚88.10117244720459" proc.memory.rssMB‚ 3001.5703125* proc.memory.percent‚18.320131301879883! disk.sdb2.out‚817.94287109375 disk.dm-0.in‚ 1102.390625 disk.sdb.in‚ 0.92578125 disk.rbd2.in‚0 disk.sdb.out‚817.94287109375 disk.sda.out‚817.94287109375 disk.rbd0.out‚ 1.912109375# disk.nvme1n1.out‚44608.33203125 cpu‚25.22219915479457 proc.cpu.threads‚17) disk./.usagePercent‚75.85296534198268 disk.sdb2.in‚ 0.92578125 disk.sda1.out‚0 disk.md0p1.in‚5.3125) proc.memory.availableMB‚ 1949.50390625$ disk./.usageGB‚2712.869453430176 disk.dm-0.out‚ 83311.6171875# disk.nvme0n1.out‚38703.28515625 disk.sdb1.out‚0 disk.md0.in‚5.3125 disk.sdb1.in‚0 disk.nvme0n1.in‚ 603.58984375 network.sent‚ 1.5554031e+07 disk.sda1.in‚0! disk.sda2.out‚817.94287109375 disk.nvme1n1.in‚ 498.80078125 network.recv‚ 2.275523e+06 disk.md0p1.out‚ 788.16015625 disk.rbd0.in‚0 disk.md0.out‚ 788.16015625 disk.sda.in‚ 4.38671875 disk.sda2.in‚ 4.38671875®Çv”’‚(:‡ Š¹êÔ¸Ú®ÿ gpu.0.gpu‚56.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚37.0 gpu.0.powerWatts‚128.388' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚32.097 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Þè ¯p‘‚(:å Š¹êÔ蟱€ disk.nvme1n1.in‚ 498.80078125 disk.dm-0.out‚83388.06640625 disk.sda.out‚819.57080078125 disk.sda2.in‚ 4.38671875# disk.nvme1n1.out‚44608.51953125 disk.sda1.in‚0 disk.dm-0.in‚ 1102.390625 disk.nvme0n1.in‚ 603.58984375 disk.sdb.out‚819.57080078125! disk.sda2.out‚819.57080078125* proc.memory.percent‚18.320131301879883 proc.cpu.threads‚17 network.sent‚ 1.556025e+07$ disk./.usageGB‚2712.869300842285 disk.rbd0.out‚ 1.97509765625! disk.sdb2.out‚819.57080078125 disk.sda1.out‚0 disk.md0p1.in‚5.3125) proc.memory.availableMB‚ 1948.97265625" proc.memory.rssMB‚ 3001.5703125 network.recv‚ 2.280359e+06( disk./.usagePercent‚75.8529610755614 disk.sdb1.out‚0 disk.sdb.in‚ 0.92578125 disk.rbd0.in‚0 disk.sdb1.in‚0 disk.md0.out‚ 789.7265625 disk.sda.in‚ 4.38671875$ memory_percent‚88.10441493988037 disk.rbd2.in‚0 disk.md0p1.out‚ 789.7265625 cpu‚25.22176851105635 disk.md0.in‚5.3125 disk.rbd2.out‚ 2.06201171875 disk.sdb2.in‚ 0.92578125! disk.nvme0n1.out‚ 38779.546875‘Ï[2’- _wandbruntime‚7151  _runtime‚7151¬‡1iÅ“¿ % lr-AdamW/pg1‚0.000490606181798244 % lr-AdamW/pg2‚0.000490606181798244 trainer/global_step‚6149 % _timestamp‚1.7885175256399002e+09 _runtime‚7151.110371828 _step‚503r¤7Ê”Ä % lr-AdamW/pg2‚0.000490606181798244 trainer/global_step‚6149 % _timestamp‚1.7885175256399002e+09 _runtime‚7151.110371828 _step‚503 % lr-AdamW/pg1‚0.000490606181798244÷j1Š2•- _wandbruntime‚7151  _runtime‚7151µÉæ¬ü'–öO SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177047040 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚495276717267280 _step‚504 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411420 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _runtime‚7151.112422638 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26088 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 398533632 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 127195294 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7308149047808 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚502249765119844 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚506626673355264 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7308359033344 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007241 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚598 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚502250679570624 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚513626728155648 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407872 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚37008696 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 % _timestamp‚1.7885175256416788e+09 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 281605448 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407872 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410816 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6999137389568 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 116101347 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 154410154 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚513625810744832 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955591168 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚506626691358208 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 281631538 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6973047852564 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚79092055 trainer/global_step‚6149 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6999137389568 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 116101347 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 154410154 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚513625810744832 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955591168 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚506626691358208 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 281631538 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6973047852564 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚79092055 trainer/global_step‚6149 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚502249765119844 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚506626673355264 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7308359033344 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007241 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984ø˜æÝÂ2˜- _wandbruntime‚7152  _runtime‚7152.t=¥™€N WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚513707471267840 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6974156513882 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 281676053 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 250689536 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _runtime‚7152.328128497 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7309306839552 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450988 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 154460859 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚513707471267840 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177047040 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007241 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 116119340 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 % _timestamp‚1.7885175268556457e+09 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 281676053 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955591168 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚513706553843712 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7001149614080 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚506706321653760 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1187 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚506706303650816 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 116118746 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7000250192896 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚574198967222272 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚566889409693184 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚495354714400768 _step‚505 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚37014317 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26114 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899421184 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411628 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 281649937 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31686 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚37014909 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚502328853875290 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 154434745 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917424128 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899421184 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 127215194 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚506706321653760 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚506706303650816 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 127215192 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚79104429 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7000250192896 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 127215194 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177047040 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7309533588992 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6975053925510 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚566889433633280 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 trainer/global_step‚6149 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0¡L™\ 'š…N N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚37014317 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26114 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899421184 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411628 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 281649937 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31686 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚37014909 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚502328853875290 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 154434745 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917424128 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899421184 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 127215194 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚506706321653760 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚506706303650816 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 127215192 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚79104429 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7000250192896 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 127215194 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177047040 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7309533588992 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6975053925510 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚566889433633280 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 trainer/global_step‚6149 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚513707471267840 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6974156513882 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 281676053 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 250689536 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _runtime‚7152.328128497 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7309306839552 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450988 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 154460859 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚513707471267840 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177047040 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007241 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 116119340 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 % _timestamp‚1.7885175268556457e+09 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 281676053 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955591168 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚513706553843712 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7001149614080 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚506706321653760 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1187 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚506706303650816 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 116118746 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7000250192896 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚574198967222272 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚566889409693184 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚495354714400768 _step‚505ùì‘Dˆ2›- _wandbruntime‚7152  _runtime‚7152 ìv]œ× 3loss/FBt3/assignment_loss‚0.0020316855834401145 /loss/FBt1/detection_loss‚2.8659698605537414 .loss/FRt1/detection_loss‚2.010587629561836 /loss/FRt3/detection_loss‚2.3120143569328566 epoch‚5 trainer/global_step‚6149 /loss/FRt3/assignment_loss‚2.180937585505572 0loss/SRqqt4/detection_loss‚3.155979225039482 /loss/FBt3/detection_loss‚2.3090586445548316 .loss/FBt4/detection_loss‚3.196311483780543 2loss/SRqqt1/assignment_loss‚1.1835674432034675 2loss/SRqqt3/assignment_loss‚1.2290865736082197 1loss/FBt4/assignment_loss‚0.17057481298653934 /loss/FRt4/detection_loss‚2.8003425444875445 _step‚506 .loss/FRt2/assignment_loss‚1.12121129155159 2loss/SRqqt4/assignment_loss‚2.4765591830015183 1loss/SRqqt3/detection_loss‚1.8394330471754075 % _timestamp‚1.7885175268603299e+09 0loss/FRt1/assignment_loss‚2.5285993941494675 1loss/FBt2/assignment_loss‚0.18948906301242954 1loss/SRqqt1/detection_loss‚1.4861123117116781 1loss/SRqqt2/detection_loss‚0.9998730410422597 %loss/total_loss‚40.33353390481581 2loss/SRqqt2/assignment_loss‚1.0830791113632066 -loss/FBt2/detection_loss‚0.73740677209571 _runtime‚7152.330625148 /loss/FRt4/assignment_loss‚3.021450444630214 3loss/FBt1/assignment_loss‚0.0002752048198090051 .loss/FRt2/detection_loss‚1.433583194464445Š W~bÜ 3loss/FBt3/assignment_loss‚0.0020316855834401145 /loss/FBt1/detection_loss‚2.8659698605537414 .loss/FRt1/detection_loss‚2.010587629561836 /loss/FRt3/detection_loss‚2.3120143569328566 epoch‚5 trainer/global_step‚6149 /loss/FRt3/assignment_loss‚2.180937585505572 0loss/SRqqt4/detection_loss‚3.155979225039482 /loss/FBt3/detection_loss‚2.3090586445548316 .loss/FBt4/detection_loss‚3.196311483780543 2loss/SRqqt1/assignment_loss‚1.1835674432034675 2loss/SRqqt3/assignment_loss‚1.2290865736082197 1loss/FBt4/assignment_loss‚0.17057481298653934 /loss/FRt4/detection_loss‚2.8003425444875445 _step‚506 .loss/FRt2/assignment_loss‚1.12121129155159 2loss/SRqqt4/assignment_loss‚2.4765591830015183 1loss/SRqqt3/detection_loss‚1.8394330471754075 % _timestamp‚1.7885175268603299e+09 0loss/FRt1/assignment_loss‚2.5285993941494675 1loss/FBt2/assignment_loss‚0.18948906301242954 1loss/SRqqt1/detection_loss‚1.4861123117116781 1loss/SRqqt2/detection_loss‚0.9998730410422597 %loss/total_loss‚40.33353390481581 2loss/SRqqt2/assignment_loss‚1.0830791113632066 -loss/FBt2/detection_loss‚0.73740677209571 _runtime‚7152.330625148 /loss/FRt4/assignment_loss‚3.021450444630214 3loss/FBt1/assignment_loss‚0.0002752048198090051 .loss/FRt2/detection_loss‚1.433583194464445ú¯VV—“ž‚(:ˆ ™¹êÔøäê gpu.0.gpu‚2.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚37.0 gpu.0.powerWatts‚155.067' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚38.76675 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0^àkŸ‚(:à ™¹êÔ¾ßǃ disk.rbd2.in‚0 disk.nvme0n1.in‚ 603.58984375 proc.cpu.threads‚17# disk.nvme1n1.out‚44608.76171875 network.recv‚ 2.283622e+06 disk.sdb.out‚821.78955078125 disk.sda.out‚821.78955078125! disk.sda2.out‚821.78955078125 disk.md0.in‚5.3125 disk.rbd2.out‚2.125! disk.sdb2.out‚821.78955078125# disk./.usageGB‚2712.99458694458 disk.sdb2.in‚ 0.92578125 cpu‚25.22121229550145 network.sent‚ 1.5565527e+07 disk.sda.in‚ 4.38671875 disk.rbd0.in‚0 disk.rbd0.out‚ 1.97509765625 disk.sda1.in‚0 disk.dm-0.out‚ 83512.9921875( proc.memory.availableMB‚ 1949.3671875 disk.nvme1n1.in‚ 498.80078125 disk.sdb1.out‚0 disk.dm-0.in‚ 1102.390625 disk.md0p1.in‚5.3125 disk.sdb.in‚ 0.92578125 disk.sda1.out‚0 disk.md0p1.out‚ 791.87109375# disk.nvme0n1.out‚38904.23046875$ memory_percent‚88.10200691223145" proc.memory.rssMB‚ 3001.5703125* proc.memory.percent‚18.320131301879883) disk./.usagePercent‚75.85646412741787 disk.sda2.in‚ 4.38671875 disk.sdb1.in‚0 disk.md0.out‚ 791.87109375YŒ§c‘ ‚(:† ¨¹êÔ–ž³ÿ gpu.0.gpu‚6.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚36.0 gpu.0.powerWatts‚131.79' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚32.9475 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0S¶»ùg¡‚(:Ü ¨¹êÔ®–  network.sent‚ 1.5675802e+07 disk.md0.out‚ 793.3203125 disk.sda1.in‚0 disk.nvme0n1.in‚ 603.58984375$ memory_percent‚88.10360431671143 disk.sdb.in‚ 0.92578125 disk.rbd0.out‚ 1.97509765625* proc.memory.percent‚18.320131301879883 cpu‚25.2208723573657$ disk./.usageGB‚2713.119243621826 disk.md0.in‚5.3125 disk.rbd2.in‚0 disk.rbd2.out‚2.125 disk.dm-0.out‚ 83689.9609375 disk.md0p1.out‚ 793.3203125" proc.memory.rssMB‚ 3001.5703125 network.recv‚ 2.289381e+06 disk.sdb1.out‚0" disk.nvme0n1.out‚ 39080.9765625 disk.sda.in‚ 4.38671875! disk.nvme1n1.out‚ 44608.984375 proc.cpu.threads‚17 disk.sda.out‚823.30322265625 disk.sdb2.in‚ 0.92578125! disk.sda2.out‚823.30322265625) disk./.usagePercent‚75.85994958028654 disk.sdb1.in‚0 disk.dm-0.in‚ 1102.390625 disk.md0p1.in‚5.3125 disk.nvme1n1.in‚ 498.80078125! disk.sdb2.out‚823.30322265625 disk.sda1.out‚0 disk.sdb.out‚823.30322265625 disk.sda2.in‚ 4.38671875 disk.rbd0.in‚0) proc.memory.availableMB‚ 1949.10546875·¸”¢‚(:‰ ·¹êÔ®çÿ gpu.0.gpu‚53.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚37.0 gpu.0.powerWatts‚119.091' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚29.77275 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0©â]n£‚(:ã ·¹êÔÊô‚€ disk.sdb1.in‚0 network.sent‚ 1.5685382e+07 disk.rbd0.in‚0# disk.nvme1n1.out‚44609.31640625 disk.rbd2.in‚0 disk.dm-0.in‚ 1102.390625) proc.memory.availableMB‚ 1948.83984375 disk.md0p1.out‚ 794.87109375" disk.nvme0n1.out‚ 39186.0546875 disk.sdb.out‚824.90380859375 cpu‚25.220456502020284 network.recv‚ 2.300197e+06 disk.sda.in‚ 4.38671875 disk.sdb1.out‚0 disk.md0.in‚5.3125 disk.sda1.out‚0 disk.md0p1.in‚5.3125* proc.memory.percent‚18.320131301879883$ disk./.usageGB‚2713.369300842285 disk.rbd0.out‚ 1.97509765625 disk.md0.out‚ 794.87109375 disk.sdb2.in‚ 0.92578125! disk.sdb2.out‚824.90380859375 proc.cpu.threads‚17) disk./.usagePercent‚75.86694128482765 disk.sda.out‚824.90380859375 disk.nvme1n1.in‚ 498.80078125 disk.nvme0n1.in‚ 603.58984375$ memory_percent‚88.10522556304932 disk.rbd2.out‚2.125 disk.sda1.in‚0 disk.dm-0.out‚83795.37109375 disk.sdb.in‚ 0.92578125" proc.memory.rssMB‚ 3001.5703125 disk.sda2.in‚ 4.38671875! disk.sda2.out‚824.90380859375Ƞ蔤‚(:‰ ƹêÔàìÛÿ gpu.0.gpu‚46.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚168.213' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.05325 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0§Õ2êc¥‚(:Ø ƹêÔضàÿ network.recv‚ 2.304791e+06) disk./.usagePercent‚75.86693968491967$ disk./.usageGB‚2713.369243621826 disk.nvme1n1.in‚ 498.80078125! disk.sdb2.out‚826.37939453125) proc.memory.availableMB‚ 1949.21484375 cpu‚25.220104704781853 proc.cpu.threads‚17 disk.sda2.in‚ 4.38671875 disk.rbd0.in‚0" disk.nvme1n1.out‚ 44609.5078125 disk.rbd2.out‚2.125 disk.dm-0.out‚ 83867.4453125$ memory_percent‚88.10293674468994 disk.md0p1.out‚ 796.28125 disk.sda.in‚ 4.38671875 disk.md0.in‚5.3125 disk.sdb2.in‚ 0.92578125 disk.sda.out‚826.37939453125 disk.md0.out‚ 796.28125 disk.sda1.in‚0 disk.nvme0n1.out‚ 39257.9375" proc.memory.rssMB‚ 3001.5703125! disk.sda2.out‚826.37939453125 disk.rbd2.in‚0 disk.md0p1.in‚5.3125* proc.memory.percent‚18.320131301879883 disk.rbd0.out‚ 1.97509765625 disk.sdb1.in‚0 disk.dm-0.in‚ 1102.390625 disk.nvme0n1.in‚ 603.58984375 network.sent‚ 1.5689457e+07 disk.sdb.in‚ 0.92578125 disk.sdb.out‚826.37939453125 disk.sdb1.out‚0 disk.sda1.out‚0!%sê2¦- _wandbruntime‚7209  _runtime‚7209^IoÀƧÀ & lr-AdamW/pg1‚0.0004844735439221388 & lr-AdamW/pg2‚0.0004844735439221388 trainer/global_step‚6199 % _timestamp‚1.7885175845119233e+09 _runtime‚ 7209.98392009 _step‚507|í6(˨Š% _timestamp‚1.7885175845119233e+09 _runtime‚ 7209.98392009 _step‚507 & lr-AdamW/pg1‚0.0004844735439221388 & lr-AdamW/pg2‚0.0004844735439221388 trainer/global_step‚6199ûĉþQ2©- _wandbruntime‚7209  _runtime‚7209ÝðøAªöO GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚79710855 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407872 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚571266075449344 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 283833250 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 128190196 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 155669144 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 trainer/global_step‚6199 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚517663897513472 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917410816 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚571266051509248 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007241 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7054791969792 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 117002784 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚79710853 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚578632276330496 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 155643056 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚517663897513472 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚499175721973680 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177047040 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410816 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 128190194 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7055691377664 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 259091456 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚506204216483352 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214665 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7028494509672 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 117002176 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚517662980102656 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 128190196 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 235151360 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚510608206135808 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7054791969792 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955591168 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚37291929 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚606 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 _runtime‚7209.990219548 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚510608188132864 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407872 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚510608188132864 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450780 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26088 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 128190194 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚499175739013040 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411420 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984oßî(«ûO YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚517663897513472 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚499175721973680 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177047040 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410816 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 128190194 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7055691377664 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 259091456 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚506204216483352 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214665 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7028494509672 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 117002176 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚517662980102656 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 128190196 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 235151360 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚510608206135808 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7054791969792 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955591168 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚37291929 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚606 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 _runtime‚7209.990219548 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚510608188132864 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407872 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚510608188132864 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450780 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26088 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 128190194 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚499175739013040 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411420 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚79710855 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407872 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚571266075449344 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 283833250 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 128190196 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 155669144 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 trainer/global_step‚6199 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚517663897513472 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917410816 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚571266051509248 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007241 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7054791969792 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 117002784 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚79710853 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚578632276330496 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 155643056 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚510687836431360 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚587 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7055904773120 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚578722753109504 _runtime‚7211.161136472 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 _step‚509 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 155667647 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411628 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26114 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 128210094 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 155693761 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 283903855 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 398533632 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7030500582618 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007241 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 283877739 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚510687818428416 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 246495232 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955591168 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7055904773120 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 128210092 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚79723231 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚571355394670080 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 128210094 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899421184 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚499253719107168 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917424128 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177047040 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚37296990 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚517744640625664 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7367135884288 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚510687818428416 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26114 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 222555136 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450988 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚510687836431360 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚79723229 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚589 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7367358439424 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31686 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚517743723201536 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 283903855 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 155693761 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚578722506614272 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚506283305238798 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 155667647 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚37297577 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 trainer/global_step‚6199 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7029603170990 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚499253702067808 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 117020219 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1187 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564@A¸é®…N PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450988 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚510687836431360 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚79723229 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚589 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7367358439424 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31686 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚517743723201536 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 283903855 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 155693761 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚578722506614272 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚506283305238798 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 155667647 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚37297577 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 trainer/global_step‚6199 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7029603170990 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚499253702067808 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 117020219 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1187 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚510687836431360 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚587 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7055904773120 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚578722753109504 _runtime‚7211.161136472 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 _step‚509 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 155667647 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411628 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26114 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 128210094 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 155693761 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 283903855 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 398533632 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7030500582618 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007241 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 283877739 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚510687818428416 ICDeviceStatsMonitor.on^Eæ@" _train_batch_end/allocation.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 246495232 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955591168 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7055904773120 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 128210092 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚79723231 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚571355394670080 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 128210094 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899421184 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚499253719107168 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917424128 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177047040 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚37296990 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚517744640625664 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7367135884288 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚510687818428416 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26114 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 222555136 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 _step‚512 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7110446550016 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 129185096 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚514589702910464 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7110446550016 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7111345957888 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 129185096 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 398533632 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚503074726680080 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚521701066871296 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410816 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚521700149460480 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 156875958 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 129185094 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚601 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214665 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407872 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 156902046 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 117903730 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955591168 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 129185094 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955591168 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450780 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚503074743719440 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚583155802940928 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚583156038963712 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26088 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 117903129 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 212082688 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _runtime‚7269.960709703 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚514589720913408 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 286061052 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚80329655 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚80329653 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1187 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917410816 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26088 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚514589702910464 % _timestamp‚1.7885176444900906e+09 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 236022784 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚599 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7083941166780 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚521701066871296 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007241 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚510158667846860 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7084838578200 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 trainer/global_step‚6249 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚521700149460480 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚510159582297640 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407872 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 286087142 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚37574075 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚575732012546048 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7423790394880 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411420 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚514589720913408 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7424002477568 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚521701066871296 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007241 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚510158667846860 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7084838578200 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 trainer/global_step‚6249 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚521700149460480 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚510159582297640 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407872 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 286087142 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚37574075 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚575732012546048 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7423790394880 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411420 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚514589720913408 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7424002477568 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 _step‚512 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7110446550016 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 129185096 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚514589702910464 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7110446550016 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7111345957888 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 129185096 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 398533632 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚503074726680080 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚521701066871296 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410816 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚521700149460480 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 156875958 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 129185094 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚601 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214665 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407872 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 156902046 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 117903730 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955591168 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 129185094 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955591168 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450780 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚503074743719440 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚583155802940928 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚583156038963712 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26088 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 117903129 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 212082688 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _runtime‚7269.960709703 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚514589720913408 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 286061052 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚80329655 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚80329653 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1187 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917410816 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26088 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚514589702910464€Z@…/2À- _wandbruntime‚7271  _runtime‚7271‚Ñ…è_ÁÿM JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177047040 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 \JDeviceStatsMoni¢ •ñ¦%tor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7112458774528 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899421184 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7425179550208 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7424959092224 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚575821355706880 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 129204994 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 129204992 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚514669333206016 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 117921715 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7085049828098 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007241 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7112458774528 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 156900549 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 286105541 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 286131657 % _timestamp‚1.7885176456811295e+09 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 156900549 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚601 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 244398080 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177047040 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 129204994 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚37579083 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7085947239726 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 _runtime‚ 7271.15235943 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 129204992 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚521781809983488 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 117921112 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 trainer/global_step‚6249 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7111559353344 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚514669351208960 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚80342029 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899421184 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26114 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚583246290859008 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚503152706774208 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 398533632 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚575821331766784 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 286131657 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917424128 _step‚513 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411628³$LÚ '„N HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 _runtime‚ 7271.15235943 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 129204992 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚521781809983488 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 117921112 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 trainer/global_step‚6249 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7111559353344 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚514669351208960 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚80342029 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899421184 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26114 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚583246290859008 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚503152706774208 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 398533632 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚575821331766784 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 286131657 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917424128 _step‚513 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411628 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177047040 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7112458774528 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899421184 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7425179550208 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7424959092224 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚575821355706880 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 129204994 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 129204992 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚514669333206016 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 117921715 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7085049828098 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007241 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7112458774528 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 156900549 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 286105541 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 286131657 % _timestamp‚1.7885176456811295e+09 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 156900549 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚601 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 244398080 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177047040 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 129204994 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚37579083 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7085947239726.7e2Ã- _wandbruntime‚7271  _runtime‚7271"¤ŽZÄÔ 1loss/SRqqt1/detection_loss‚1.3279750079980919 1loss/SRqqt2/detection_loss‚1.0482112804397208 1loss/SRqqt2/assignment_loss‚0.962954848363507 0loss/FBt2/assignment_loss‚0.1343137975581506 /loss/FBt2/detection_loss‚0.6270914133521728 -loss/FBt3/detection_loss‚3.62514106290681 % _timestamp‚1.7885176456834073e+09 0loss/FRt4/assignment_loss‚3.6753600162008535 0loss/FBt1/assignment_loss‚0.2498205803295903 1loss/SRqqt3/detection_loss‚1.7090018741917192 _runtime‚7271.153591705 0loss/FBt3/assignment_loss‚0.6789152975460249 /loss/FRt3/detection_loss‚2.5338113758909073 /loss/FBt1/detection_loss‚2.2540186166763307 /loss/FRt1/detection_loss‚1.7842360234032497 /loss/FRt2/detection_loss‚1.7562470455702983 1loss/SRqqt4/detection_loss‚2.5232947702918733 /loss/FBt4/detection_loss‚2.9529178738594055 trainer/global_step‚6249 _step‚514 0loss/FRt1/assignment_loss‚1.8554051068791173 /loss/FRt3/assignment_loss‚2.638408987145675 epoch‚5 0loss/FRt2/assignment_loss‚1.2554656727272167 2loss/SRqqt4/assignment_loss‚1.5179201598678316 0loss/FBt4/assignment_loss‚0.8540681534297099 .loss/FRt4/detection_loss‚3.411225440709487 %loss/total_loss‚42.04463278760914 2loss/SRqqt1/assignment_loss‚1.2161303382499942 2loss/SRqqt3/assignment_loss‚1.4526980440214015…On©_ÅÙ /loss/FRt3/detection_loss‚2.5338113758909073 /loss/FBt1/detection_loss‚2.2540186166763307 /loss/FRt1/detection_loss‚1.7842360234032497 /loss/FRt2/detection_loss‚1.7562470455702983 1loss/SRqqt4/detection_loss‚2.5232947702918733 /loss/FBt4/detection_loss‚2.9529178738594055 trainer/global_step‚6249 _step‚514 0loss/FRt1/assignment_loss‚1.8554051068791173 /loss/FRt3/assignment_loss‚2.638408987145675 epoch‚5 0loss/FRt2/assignment_loss‚1.2554656727272167 2loss/SRqqt4/assignment_loss‚1.5179201598678316 0loss/FBt4/assignment_loss‚0.8540681534297099 .loss/FRt4/detection_loss‚3.411225440709487 %loss/total_loss‚42.04463278760914 2loss/SRqqt1/assignment_loss‚1.2161303382499942 2loss/SRqqt3/assignment_loss‚1.4526980440214015 1loss/SRqqt1/detection_loss‚1.3279750079980919 1loss/SRqqt2/detection_loss‚1.0482112804397208 1loss/SRqqt2/assignment_loss‚0.962954848363507 0loss/FBt2/assignment_loss‚0.1343137975581506 /loss/FBt2/detection_loss‚0.6270914133521728 -loss/FBt3/detection_loss‚3.62514106290681 % _timestamp‚1.7885176456834073e+09 0loss/FRt4/assignment_loss‚3.6753600162008535 0loss/FBt1/assignment_loss‚0.2498205803295903 1loss/SRqqt3/detection_loss‚1.7090018741917192 _runtime‚7271.153591705 0loss/FBt3/assignment_loss‚0.6789152975460249‚4¾Ÿy“Æ‚(:ˆ ‘ºêԕײÿ gpu.0.gpu‚16.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚37.0 gpu.0.powerWatts‚158.746' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚39.6865 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¦E©”fÇ‚(:Û ‘ºêÔì˜Ï€ disk.sda2.in‚ 4.38671875 disk.md0.out‚ 804.55078125 disk.sda1.out‚0$ memory_percent‚92.06981658935547 proc.cpu.threads‚17 disk.nvme0n1.in‚ 603.58984375 disk.sdb.out‚834.98779296875 disk.sda.out‚834.98779296875 disk.nvme1n1.in‚ 498.80078125 disk.sda1.in‚0 disk.md0p1.in‚5.3125 disk.md0p1.out‚ 804.55078125& proc.memory.availableMB‚ 1299.28125 disk.dm-0.in‚ 1102.390625 disk.dm-0.out‚ 84503.8359375 network.recv‚ 2.333242e+06 disk.sdb.in‚ 0.92578125 disk.rbd2.in‚0 disk.rbd2.out‚2.125 disk.sdb2.in‚ 0.92578125" proc.memory.rssMB‚ 3001.5703125% disk./.usageGB‚2713.8696517944336* proc.memory.percent‚18.320131301879883! disk.sda2.out‚834.98779296875 disk.rbd0.out‚ 1.97509765625 network.sent‚ 1.582392e+07) disk./.usagePercent‚75.88093130686286 disk.md0.in‚5.3125 disk.sda.in‚ 4.38671875 disk.rbd0.in‚0 disk.sdb1.out‚0 cpu‚25.218042597269108 disk.nvme0n1.out‚ 39893.3125" disk.nvme1n1.out‚ 44610.5234375 disk.sdb1.in‚0! disk.sdb2.out‚834.98779296875oa=’È‚(:‡  ºêÔ·´Áÿ gpu.0.gpu‚54.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚131.79' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚32.9475 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0º¤X}iÉ‚(:Þ  ºêÔêÓôÿ disk.md0p1.out‚ 806.2421875" disk.nvme0n1.out‚ 40158.3984375 disk.sdb.out‚836.74658203125 network.sent‚ 1.5931879e+07 disk.sda.out‚836.74658203125$ memory_percent‚92.06981658935547 disk.sda.in‚ 4.38671875! disk.sda2.out‚836.74658203125 disk.sdb1.in‚0! disk.sdb2.out‚836.74658203125& proc.memory.availableMB‚ 1299.28125" proc.memory.rssMB‚ 3001.5703125* proc.memory.percent‚18.320131301879883 disk.sdb2.in‚ 0.92578125 disk.dm-0.out‚84769.12109375 disk.rbd0.in‚0 disk.md0.out‚ 806.2421875 disk.rbd2.in‚0 disk.rbd2.out‚2.125 disk.sdb1.out‚0 disk.dm-0.in‚ 1102.390625 proc.cpu.threads‚17# disk.nvme1n1.out‚44610.72265625 network.recv‚ 2.339349e+06 disk.md0.in‚5.3125 disk.sda1.in‚0 disk.sdb.in‚ 0.92578125( disk./.usagePercent‚75.8809284270285 disk.md0p1.in‚5.3125 disk.sda1.out‚0 disk.nvme0n1.in‚ 603.58984375% disk./.usageGB‚2713.8695487976074 disk.sda2.in‚ 4.38671875 disk.rbd0.out‚ 1.97509765625 disk.nvme1n1.in‚ 498.80078125 cpu‚25.217566364049446}#\¾“Ê‚(:ˆ ¯ºêÔ…®°ÿ gpu.0.gpu‚55.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚167.774' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚41.9435 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0“ C]Ë‚(:Ò ¯ºêÔܽæÿ disk.sda1.out‚0 disk.dm-0.in‚ 1102.390625 disk.sdb1.in‚0 network.recv‚ 2.342934e+06) disk./.usagePercent‚75.88093184016552 disk.md0.in‚5.3125 disk.rbd2.out‚2.125 disk.sdb.in‚ 0.92578125 disk.sda2.out‚ 838.173828125 disk.md0p1.in‚5.3125 disk.sda.in‚ 4.38671875 disk.md0.out‚ 807.609375( proc.memory.availableMB‚ 1299.2890625! disk.nvme0n1.out‚ 40411.765625 disk.rbd2.in‚0 disk.nvme0n1.in‚ 603.58984375# disk./.usageGB‚2713.86967086792" disk.nvme1n1.out‚ 44610.9140625 disk.md0p1.out‚ 807.609375" proc.memory.rssMB‚ 3001.5703125 disk.sdb2.in‚ 0.92578125 disk.sda1.in‚0 disk.sdb.out‚ 838.173828125* proc.memory.percent‚18.320131301879883 cpu‚25.217121956029786 disk.sda.out‚ 838.173828125 disk.sda2.in‚ 4.38671875 disk.nvme1n1.in‚ 498.80078125 disk.dm-0.out‚ 85022.6796875$ memory_percent‚92.06976890563965 proc.cpu.threads‚17 network.sent‚ 1.5939878e+07 disk.rbd0.in‚0 disk.rbd0.out‚ 1.97509765625 disk.sdb1.out‚0 disk.sdb2.out‚ 838.173828125©LŒ¸Ì‚(:… ¾ºêÔݪ¶ÿ gpu.0.gpu‚47.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚37.0 gpu.0.powerWatts‚125.88' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚31.47 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0þJölbÍ‚(:× ¾ºêÔ¨ðÕ€( proc.memory.availableMB‚ 1299.0390625 disk.md0p1.in‚5.3125 disk.sdb.in‚ 0.92578125 disk.nvme1n1.in‚ 498.80078125 disk.nvme1n1.out‚ 44611.0625 disk.sdb2.out‚840.3154296875# disk./.usageGB‚2713.86967086792 disk.sdb.out‚840.3154296875 disk.sda1.in‚0 disk.dm-0.in‚ 1102.390625 disk.nvme0n1.in‚ 603.58984375 disk.sda.in‚ 4.38671875 disk.sda2.out‚840.3154296875" proc.memory.rssMB‚ 3001.5703125* proc.memory.percent‚18.320131301879883 proc.cpu.threads‚17 network.sent‚ 1.5946142e+07) disk./.usagePercent‚75.88093184016552 disk.md0p1.out‚ 809.68359375" disk.nvme0n1.out‚ 40423.5703125 disk.sda.out‚840.3154296875 disk.rbd0.out‚ 1.97509765625 disk.sdb1.out‚0 cpu‚25.216774474476875 disk.md0.in‚5.3125 disk.sdb1.in‚0 disk.md0.out‚ 809.68359375# memory_percent‚92.0712947845459 disk.dm-0.out‚ 85034.6328125 disk.sda2.in‚ 4.38671875 disk.rbd2.out‚2.125 disk.sdb2.in‚ 0.92578125 network.recv‚ 2.347706e+06 disk.rbd0.in‚0 disk.rbd2.in‚0 disk.sda1.out‚0ä /{2Î- _wandbruntime‚7331  _runtime‚7331Í—ÎÇÏÁ & lr-AdamW/pg1‚0.0004721747732163208 & lr-AdamW/pg2‚0.0004721747732163208 trainer/global_step‚6299 % _timestamp‚1.7885177058614461e+09 _runtime‚7331.331757839 _step‚515Û;®ÌÐÆ _runtime‚7331.331757839 _step‚515 & lr-AdamW/pg1‚0.0004721747732163208 & lr-AdamW/pg2‚0.0004721747732163208 trainer/global_step‚6299 % _timestamp‚1.7885177058614461e+09ƒ||X2Ñ- _wandbruntime‚7331  _runtime‚7331.GÃtÒöO H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214665 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 130179994 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177047040 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚525737318818304 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7166101130240 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 222568448 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955591168 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 158108860 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7167000538112 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚525737318818304 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7166101130240 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 158134948 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 118805380 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚37856327 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚598 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 M=DeviceStatsMonitor.on_train_batch_sta›$&êârt/active_bytes.all.peak‚ 13955591168 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚587679991175680 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7139387823888 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 288314944 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1187 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 % _timestamp‚1.7885177058621528e+09 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26090 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407872 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26088 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚80948455 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7482240161280 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚506973731386480 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚525738236229120 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 130179994 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 _step‚516 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7482017592832 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚600 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 398533632 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450780 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407872 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚518571217688064 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 158108860 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1212 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 130179996 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚518571235691008 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7167000538112 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7140285235308 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26088 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚525738236229120 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚37856925 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚580197997522944 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214665 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 130179994 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177047040 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚525737318818304 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7166101130240 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 222568448 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955591168 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 158108860 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7167000538112 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚525737318818304 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7166101130240 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 158134948 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 118805380 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚37856327 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚598 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955591168 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚587679991175680 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7139387823888 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607„ù±2Ô- _wandbruntime‚7332  _runtime‚7332/÷¾ª'Õ€N L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 288359459 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7141393896834 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 158133451 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚525818979341312 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7167213933568 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31686 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚580287316743680 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26114 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450988 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 130199894 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 158159565 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚518650847983616 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚507051728519968 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 _runtime‚7332.307233833 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 288359459 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚612 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚587770480195072 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 288333343 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7483409946624 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411628 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007241 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 130199892 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 118822717 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚507051711480608 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚518650865986560 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899421184 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 158133451 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 398533632 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 288333343 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7483187391488 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚514193122416802 _step‚517 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 130199894 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 118823331 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 288359459 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7141393896834 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 158133451 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚525818979341312 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7167213933568 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31686 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚580287316743680 UHDeviceStatsMonitor.on_trainn¡…I_batch_end/requested_bytes.large_pool.current‚17039360 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26114 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450988 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 130199894 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 158159565 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚518650847983616 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚507051728519968 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 _runtime‚7332.307233833 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 288359459 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚612 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚587770480195072 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 288333343 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7483409946624 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411628 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007241 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 130199892 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 118822717 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚507051711480608 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚518650865986560 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899421184 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 158133451 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 398533632 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 288333343 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7483187391488 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚514193122416802 _step‚517 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 130199894 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1…äŽû2×- _wandbruntime‚7332  _runtime‚7332ç‚ Ž_ØÙ 2loss/SRqqt2/assignment_loss‚0.8962441256735474 1loss/FBt2/assignment_loss‚0.08509034034771408 2loss/SRqqt1/assignment_loss‚1.3184473176837945 1loss/FBt3/assignment_loss‚0.15057547075792338 0loss/FBt4/assignment_loss‚0.2506141613318635 /loss/FRt2/detection_loss‚1.2979950875526778 0loss/SRqqt4/detection_loss‚2.330519724276758 0loss/FRt3/assignment_loss‚2.6939792960882185 0loss/FRt4/assignment_loss‚3.2433129847049713 % _timestamp‚1.7885177068381755e+09 /loss/FRt3/detection_loss‚2.2447422979399563 trainer/global_step‚6299 _step‚518 2loss/SRqqt3/assignment_loss‚1.2267816522914283 2loss/FBt1/assignment_loss‚0.042675209103617814 1loss/SRqqt1/detection_loss‚1.3806551029203369 1loss/SRqqt3/detection_loss‚1.7141499485609666 .loss/FBt4/detection_loss‚2.581595855099814 _runtime‚7332.308283057 0loss/FRt2/assignment_loss‚0.9523244809221338 .loss/FRt1/detection_loss‚1.684271526372473 /loss/FRt4/detection_loss‚3.4403452277183533 /loss/FBt2/detection_loss‚0.6342557458864415 2loss/SRqqt4/assignment_loss‚1.7546800084171756 1loss/SRqqt2/detection_loss‚0.8452919418923557 epoch‚5 /loss/FRt1/assignment_loss‚2.534406948764369 /loss/FBt1/detection_loss‚2.4501180231571196 /loss/FBt3/detection_loss‚3.4512057196010244 %loss/total_loss‚39.20427819706504}½¹šdÙÞ 2loss/SRqqt3/assignment_loss‚1.2267816522914283 2loss/FBt1/assignment_loss‚0.042675209103617814 1loss/SRqqt1/detection_loss‚1.3806551029203369 1loss/SRqqt3/detection_loss‚1.7141499485609666 .loss/FBt4/detection_loss‚2.581595855099814 _runtime‚7332.308283057 0loss/FRt2/assignment_loss‚0.9523244809221338 .loss/FRt1/detection_loss‚1.684271526372473 /loss/FRt4/detection_loss‚3.4403452277183533 /loss/FBt2/detection_loss‚0.6342557458864415 2loss/SRqqt4/assignment_loss‚1.7546800084171756 1loss/SRqqt2/detection_loss‚0.8452919418923557 epoch‚5 /loss/FRt1/assignment_loss‚2.534406948764369 /loss/FBt1/detection_loss‚2.4501180231571196 /loss/FBt3/detection_loss‚3.4512057196010244 %loss/total_loss‚39.20427819706504 2loss/SRqqt2/assignment_loss‚0.8962441256735474 1loss/FBt2/assignment_loss‚0.08509034034771408 2loss/SRqqt1/assignment_loss‚1.3184473176837945 1loss/FBt3/assignment_loss‚0.15057547075792338 0loss/FBt4/assignment_loss‚0.2506141613318635 /loss/FRt2/detection_loss‚1.2979950875526778 0loss/SRqqt4/detection_loss‚2.330519724276758 0loss/FRt3/assignment_loss‚2.6939792960882185 0loss/FRt4/assignment_loss‚3.2433129847049713 % _timestamp‚1.7885177068381755e+09 /loss/FRt3/detection_loss‚2.2447422979399563 trainer/global_step‚6299 _step‚518†,×a‡”Ú‚(:‰ ͺêÔÕí«€ gpu.0.gpu‚48.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚160.759' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚40.18975 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0k ˆo\Û‚(:Ñ ͺêÔåôê€ disk.sdb2.out‚ 841.873046875 disk.dm-0.out‚ 85034.921875 disk.nvme0n1.in‚ 603.58984375 disk.sda2.in‚ 4.38671875 network.sent‚ 1.5949238e+07 network.recv‚ 2.351089e+06 disk.nvme0n1.out‚ 40423.65625 disk.sdb.out‚ 841.873046875 disk.sda.out‚ 841.873046875 disk.sda1.out‚0" proc.memory.rssMB‚ 3001.5703125* proc.memory.percent‚18.320131301879883 disk.md0p1.in‚5.3125 disk.sda2.out‚ 841.873046875 disk.nvme1n1.in‚ 498.80078125 disk.rbd2.out‚2.125 disk.sda1.in‚0 proc.cpu.threads‚17 disk.dm-0.in‚ 1102.390625! disk.nvme1n1.out‚ 44611.265625$ memory_percent‚92.07155704498291 disk.md0p1.out‚ 811.1796875 disk.sdb.in‚ 0.92578125 disk.sda.in‚ 4.38671875 disk.rbd0.in‚0 disk.sdb2.in‚ 0.92578125$ disk./.usageGB‚2713.869728088379 disk.md0.in‚5.3125 disk.md0.out‚ 811.1796875) proc.memory.availableMB‚ 1298.99609375 disk.rbd0.out‚ 1.97509765625 disk.rbd2.in‚0( disk./.usagePercent‚75.8809334400735 disk.sdb1.in‚0 disk.sdb1.out‚0 cpu‚25.21643340605225˜Û†Ë”Ü‚(:‰ ܺêÔâ –ÿ gpu.0.gpu‚57.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚170.219' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.55475 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0“:•iÝ‚(:Þ ܺêÔÁæ½ÿ" disk.nvme1n1.out‚ 44611.4921875 disk.sdb1.in‚0 disk.dm-0.in‚ 1102.390625 disk.md0p1.in‚5.3125 disk.sdb.out‚843.68505859375! disk.sda2.out‚843.68505859375) disk./.usagePercent‚75.86612298522522 disk.rbd2.in‚0 disk.nvme0n1.out‚ 40777.28125* proc.memory.percent‚18.320131301879883 disk.sda1.out‚0 disk.nvme0n1.in‚ 603.58984375 disk.rbd0.in‚0) proc.memory.availableMB‚ 1299.24609375 disk.sdb2.in‚ 0.92578125 network.recv‚ 2.357255e+06 disk.md0.in‚5.3125 disk.sda1.in‚0 disk.dm-0.out‚ 85388.7734375 disk.sdb1.out‚0 disk.sda2.in‚ 4.38671875$ memory_percent‚92.07003116607666 network.sent‚ 1.6057907e+07% disk./.usageGB‚2713.3400344848633 disk.nvme1n1.in‚ 498.80078125 disk.sdb.in‚ 0.92578125 disk.sda.out‚843.68505859375 cpu‚25.216043295382978 proc.cpu.threads‚17 disk.md0.out‚ 812.9296875! disk.sdb2.out‚843.68505859375 disk.md0p1.out‚ 812.9296875 disk.sda.in‚ 4.38671875 disk.rbd2.out‚2.125 disk.rbd0.out‚ 1.97509765625" proc.memory.rssMB‚ 3001.5703125ï÷t“Þ‚(:ˆ ëºêÔ›µ¼ÿ gpu.0.gpu‚56.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚39.0 gpu.0.powerWatts‚172.118' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.0295 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0î_µÀ_ß‚(:Ô ëºêÔÀÚîÿ% disk./.usageGB‚2712.2854919433594 disk.dm-0.out‚85538.03515625 disk.sdb.out‚ 845.576171875 disk.sda2.out‚ 845.576171875( proc.memory.availableMB‚ 1299.4609375* proc.memory.percent‚18.320131301879883 disk.sdb.in‚ 0.92578125) disk./.usagePercent‚75.83663753440442 disk.md0.in‚5.3125 disk.rbd2.out‚2.125# memory_percent‚92.0687198638916 proc.cpu.threads‚17 disk.nvme0n1.in‚ 603.58984375 disk.sda.in‚ 4.38671875 disk.sdb1.in‚0 disk.sdb1.out‚0" proc.memory.rssMB‚ 3001.5703125 network.sent‚ 1.6063972e+07 network.recv‚ 2.361601e+06 disk.dm-0.in‚ 1102.390625 disk.md0p1.in‚5.3125 disk.sda2.in‚ 4.38671875 disk.md0p1.out‚ 814.75390625 disk.rbd0.out‚ 1.97509765625# disk.nvme1n1.out‚44611.78515625 disk.md0.out‚ 814.75390625 disk.sdb2.in‚ 0.92578125 disk.sdb2.out‚ 845.576171875 disk.sda1.in‚0 disk.rbd0.in‚0 disk.nvme1n1.in‚ 498.80078125 disk.sda1.out‚0 disk.nvme0n1.out‚40926.25 disk.sda.out‚ 845.576171875 disk.rbd2.in‚0 cpu‚25.21562938219524ìÈ)”à‚(:‰ úºêÔÍÙ—ÿ gpu.0.gpu‚13.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚177.411' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚44.35275 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0÷Æ'Zá‚(:Ï úºêÔ‘âÌÿ disk.rbd0.in‚0 disk.rbd0.out‚ 1.97509765625 disk.rbd2.out‚2.125 proc.cpu.threads‚17 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.md0p1.out‚ 815.9765625# disk.nvme0n1.out‚40926.98828125 disk.sda.in‚ 4.38671875$ memory_percent‚92.06604957580566( proc.memory.availableMB‚ 1299.8984375 disk.md0p1.in‚5.3125 disk.nvme0n1.in‚ 603.58984375 disk.sda2.out‚ 846.8671875 disk.md0.in‚5.3125" proc.memory.rssMB‚ 3001.5703125) disk./.usagePercent‚75.83491123369203 disk.dm-0.out‚85538.91015625 disk.nvme1n1.in‚ 498.80078125 disk.sda1.in‚0 disk.sda1.out‚0 cpu‚25.215262027123153 network.recv‚ 2.366765e+06 disk.dm-0.in‚ 1102.390625 disk.sda.out‚ 846.8671875 disk.md0.out‚ 815.9765625 disk.sdb2.out‚ 846.8671875 network.sent‚ 1.607231e+07% disk./.usageGB‚2712.2237510681152 disk.rbd2.in‚0* proc.memory.percent‚18.320131301879883 disk.sdb.in‚ 0.92578125 disk.sdb.out‚ 846.8671875! disk.nvme1n1.out‚ 44611.921875 disk.sdb2.in‚ 0.92578125 disk.sda2.in‚ 4.38671875͆2â- _wandbruntime‚7384  _runtime‚7384)H¿wÇãÁ % _timestamp‚1.7885177590718703e+09 _runtime‚7384.542218932 _step‚519 & lr-AdamW/pg1‚0.0004660112034319892 & lr-AdamW/pg2‚0.0004660112034319892 trainer/global_step‚6349~xyvÌäÆ _runtime‚7384.542218932 _step‚519 & lr-AdamW/pg1‚0.0004660112034319892 & lr-AdamW/pg2‚0.0004660112034319892 trainer/global_step‚6349 % _timestamp‚1.7885177590718703e+09‡ûÞ§œ2å- _wandbruntime‚7384  _runtime‚7384Rá9ü'æöO aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚584663934619648 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450780 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 159341762 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 290516656 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917410816 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7540082253312 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 _step‚520 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚38140454 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚623 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407872 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26088 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚510872736092880 trainer/global_step‚6349 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7221755710464 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26090 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 131174896 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955591168 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 159367850 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 119707084 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚518068485024656 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 290542746 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚81567253 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 119707709 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚529774488176128 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7540317404672 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 235151360 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007241 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚592204275964416 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚529775405586944 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚592204016872960 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 259091456 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚522552732465664 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7222655118336 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚518067570573876 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410816 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 131174896 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407872 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26088 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚510872736092880 trainer/global_step‚6349 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7221755710464 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26090 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 131174896 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955591168 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 159367850 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 119707084 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚518068485024656 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 290542746 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚81567253 JDeviceStatsMonitor.on_train_batch_start/activl5}e.small_pool.peak‚31686 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 119707709 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚529774488176128 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7540317404672 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 235151360 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007241 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚592204275964416 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚529775405586944 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚592204016872960 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 259091456ˆ©!‹ð2è- _wandbruntime‚7385  _runtime‚7385†ÑS™'é€N T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚529856148699136 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚518147573780310 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 159366353 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450988 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7195943142314 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917424128 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚38146154 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7222868513792 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 119725165 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚618 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 131194792 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚81579631 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7541241016320 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899421184 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177047040 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚81579629 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7541467765760 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 290587261 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚592294494856704 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 159392467 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7223767934976 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 131194794 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177047040 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 131194792 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 290587261 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚522632362761216 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚584753277780480 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚522632380764160 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 226749440 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411628 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 290561145 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚592294745546240 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 250689536 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚518146659329322 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 159366353 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚529856148699136 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 119725165 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚618 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 131194792 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚81579631 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7541241016320 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899421184 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177047040 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚81579629 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7541467765760 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 290587261 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚592294494856704 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 159392467 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7223767934976 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 131194794 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177047040 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 131194792 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 290587261 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚522632362761216 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚584753277780480 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚522632380764160 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 226749440 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411628 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 290561145 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚592294745546240 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 250689536 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚518146659329322 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 159366353 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚529856148699136 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 EDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1187 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚533812574944768 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 160574664 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚589129895656448 % _timestamp‚1.7885178132900531e+09 _step‚524 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 263285760 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚533811657533952 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚38425792 trainer/global_step‚6399 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955591168 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 292744458 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚522022936388164 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 132169794 _runtime‚7438.760468956 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 160574664 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚589129919596544 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7251178549524 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407872 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450780 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411420 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007241 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚526534265246208 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7598237625344 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚522022021937384 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26090 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚533811657533952 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚526534247243264 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7278309698560 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 160600752 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚646 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚38425146ܯ](ùûO _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7597998279680 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚82186053 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚533812574944768 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 132169796 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26088 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 132169796 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚596728157221888 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410816 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 160600752 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7277410290688 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1187 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚533812574944768 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 160574664 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚589129895656448 % _timestamp‚1.7885178132900531e+09 _step‚524 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 263285760 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚533811657533952 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚38425792 trainer/global_step‚6399 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955591168 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 292744458 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚522022936388164 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 132169794 _runtime‚7438.760468956 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 160574664 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚589129919596544 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7251178549524 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407872 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450780 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411420 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007241 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚526534265246208 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7598237625344 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚522022021937384 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26090 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚533811657533952 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚526534247243264 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7278309698560 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 160600752 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚646 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚38425146 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 120611847 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7250281138104 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚514849720893408 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 254883840 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 292788947 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7279422515200 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 132189692 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 160625369 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 132189692 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 132189694 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚596818623072768 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚596818368188928 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 132189694 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 292788947 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 160599255 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 160599255 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 230943744 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007241 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7599153311744 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7279422515200 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7278523094016 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚82198429 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚82198431 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 trainer/global_step‚6399 % _timestamp‚1.7885178142454784e+09 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26114 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899421184 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 292815063 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚522101110692830 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚ 7439.71626834 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177047040 _step‚525 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26114 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 120629268 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚522102025143818 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚526613877538816 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7251389799422 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 120629899 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚631 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411628 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚38431468 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7599384255488 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚514849720893408 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 254883840 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 292788947 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7279422515200 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 132189692 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 160625369 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 132189692 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 132189694 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚596818623072768 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚596818368188928 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 132189694 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 292788¥[[947 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 160599255 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 160599255 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 230943744 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007241 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7599153311744 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7279422515200 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7278523094016 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚82198429 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚82198431 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 trainer/global_step‚6399 % _timestamp‚1.7885178142454784e+09 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26114 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899421184 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 292815063 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚522101110692830 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚ 7439.71626834 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177047040 _step‚525 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26114 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 120629268 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚522102025143818 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚526613877538816o‘°2ý- _wandbruntime‚7439  _runtime‚7439;õµ_þÙ /loss/FRt1/assignment_loss‚2.502455638379467 /loss/FRt4/assignment_loss‚3.870399689132517 1loss/FBt2/assignment_loss‚0.08063553538126333 3loss/FBt3/assignment_loss‚0.0009179027947538998 /loss/FRt3/detection_loss‚2.3273175660520793 1loss/SRqqt4/detection_loss‚2.8415220578511557 0loss/FRt3/assignment_loss‚1.9454137250781058 /loss/FRt4/detection_loss‚3.4090998375957664 % _timestamp‚1.7885178142477427e+09 2loss/SRqqt4/assignment_loss‚1.9054134172343073 1loss/FBt4/assignment_loss‚0.06147599773455361 /loss/FRt2/detection_loss‚1.4468533794430118 /loss/FBt2/detection_loss‚0.5396205106982961 .loss/FBt3/detection_loss‚4.947103798389435 _runtime‚7439.717867287 trainer/global_step‚6399 1loss/SRqqt3/detection_loss‚1.4819270842156167 %loss/total_loss‚41.64284893668377 _step‚526 3loss/FBt1/assignment_loss‚0.0020186929283557797 1loss/SRqqt1/detection_loss‚1.9056436471184905 0loss/FRt2/assignment_loss‚1.3001075828516926 1loss/SRqqt2/assignment_loss‚0.965400200090645 1loss/SRqqt3/assignment_loss‚1.233876048255775 0loss/SRqqt2/detection_loss‚0.982197451788713 epoch‚5 2loss/SRqqt1/assignment_loss‚1.1219049989416892 /loss/FRt1/detection_loss‚2.2320478396117815 .loss/FBt1/detection_loss‚2.193585666743192 .loss/FBt4/detection_loss‚2.345910668373108‘{šÒdÿÞ 2loss/SRqqt1/assignment_loss‚1.1219049989416892 /loss/FRt1/detection_loss‚2.2320478396117815 .loss/FBt1/detection_loss‚2.193585666743192 .loss/FBt4/detection_loss‚2.345910668373108 /loss/FRt1/assignment_loss‚2.502455638379467 /loss/FRt4/assignment_loss‚3.870399689132517 1loss/FBt2/assignment_loss‚0.08063553538126333 3loss/FBt3/assignment_loss‚0.0009179027947538998 /loss/FRt3/detection_loss‚2.3273175660520793 1loss/SRqqt4/detection_loss‚2.8415220578511557 0loss/FRt3/assignment_loss‚1.9454137250781058 /loss/FRt4/detection_loss‚3.4090998375957664 % _timestamp‚1.7885178142477427e+09 2loss/SRqqt4/assignment_loss‚1.9054134172343073 1loss/FBt4/assignment_loss‚0.06147599773455361 /loss/FRt2/detection_loss‚1.4468533794430118 /loss/FBt2/detection_loss‚0.5396205106982961 .loss/FBt3/detection_loss‚4.947103798389435 _runtime‚7439.717867287 trainer/global_step‚6399 1loss/SRqqt3/detection_loss‚1.4819270842156167 %loss/total_loss‚41.64284893668377 _step‚526 3loss/FBt1/assignment_loss‚0.0020186929283557797 1loss/SRqqt1/detection_loss‚1.9056436471184905 0loss/FRt2/assignment_loss‚1.3001075828516926 1loss/SRqqt2/assignment_loss‚0.965400200090645 1loss/SRqqt3/assignment_loss‚1.233876048255775 0loss/SRqqt2/detection_loss‚0.982197451788713 epoch‚5ŽÀ÷HTl€‚(:á ¶»êÔš¢éÿ disk.nvme1n1.in‚ 498.80078125 disk.sda1.in‚0" proc.memory.rssMB‚ 3001.5703125 network.recv‚ 2.412415e+06 disk.sda2.in‚ 4.38671875 disk.rbd2.out‚2.125 disk.md0p1.out‚ 822.26953125) disk./.usagePercent‚75.84063741101951 disk.md0.in‚5.3125 disk.dm-0.in‚ 1102.390625* proc.memory.percent‚18.320131301879883 disk.sda.in‚ 4.38671875 disk.rbd0.in‚0 disk.sdb2.in‚ 0.92578125 disk.sda1.out‚0 disk.dm-0.out‚85705.44921875 cpu‚25.213530394606774 network.sent‚ 1.6223016e+07 disk.nvme0n1.in‚ 603.58984375# disk.nvme1n1.out‚44684.96484375$ memory_percent‚92.06757545471191 disk.sdb1.out‚0 disk.rbd2.in‚0 disk.md0p1.in‚5.3125% disk./.usageGB‚2712.4285469055176! disk.nvme0n1.out‚ 41020.484375 disk.sdb.out‚853.37841796875 disk.sda.out‚853.37841796875 disk.sdb1.in‚0( proc.memory.availableMB‚ 1299.6484375 proc.cpu.threads‚17 disk.sdb.in‚ 0.92578125! disk.sda2.out‚853.37841796875 disk.md0.out‚ 822.26953125! disk.sdb2.out‚853.37841796875 disk.rbd0.out‚ 2.0615234375Ú†a)“‚(:ˆ ¶»êÔé•€ gpu.0.gpu‚11.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚179.246' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚44.8115 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¬Ù]ˆ’‚‚(:‡ Å»êÔ÷ÞŽþ gpu.0.gpu‚45.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚139.528' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚34.882 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ŸHñÝ]ƒ‚(:Ò Å»êÔÉ„à€ disk.sda2.in‚ 4.38671875 disk.rbd2.in‚0 disk.sda1.out‚0 cpu‚25.213095735841954 proc.cpu.threads‚17 disk.nvme0n1.out‚41120 disk.sdb.in‚ 0.92578125 disk.rbd0.out‚ 2.12451171875 disk.md0p1.in‚5.3125 disk.sdb1.in‚0 disk.rbd2.out‚2.125 disk.sdb1.out‚0 disk.md0.in‚5.3125 network.recv‚ 2.416866e+06 disk.nvme0n1.in‚ 603.58984375 disk.nvme1n1.in‚ 498.80078125$ memory_percent‚92.06616878509521) proc.memory.availableMB‚ 1299.87890625) disk./.usagePercent‚75.84249042444407" disk.nvme1n1.out‚ 44757.3359375 disk.sdb2.in‚ 0.92578125 disk.sdb2.out‚ 854.919921875 disk.dm-0.in‚ 1102.390625 disk.dm-0.out‚ 85877.3359375" proc.memory.rssMB‚ 3001.5703125 disk.sda2.out‚ 854.919921875 disk.rbd0.in‚0 disk.md0p1.out‚ 823.76171875 disk.md0.out‚ 823.76171875 disk.sda1.in‚0 disk.sda.out‚ 854.919921875 network.sent‚ 1.6319304e+07% disk./.usageGB‚2712.4948196411133 disk.sdb.out‚ 854.919921875 disk.sda.in‚ 4.38671875* proc.memory.percent‚18.320131301879883$LǦ“„‚(:ˆ Ô»êÔø¼ƒ€ gpu.0.gpu‚55.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚164.206' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚41.0515 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0[+§‹i…‚(:Þ Ô»êÔµ“ž€ disk.sda1.in‚0 disk.dm-0.out‚85893.72265625 disk.rbd0.in‚0 disk.nvme1n1.in‚ 498.80078125" proc.memory.rssMB‚ 3001.5703125 disk.md0.in‚5.3125 disk.rbd2.in‚0 network.sent‚ 1.6323013e+07 disk.rbd2.out‚2.125 disk.rbd0.out‚ 2.12451171875! disk.sdb2.out‚856.85498046875% disk./.usageGB‚2712.6202430725098 disk.md0p1.in‚5.3125 disk.md0p1.out‚ 825.6484375 disk.sda.out‚856.85498046875 disk.sda2.in‚ 4.38671875 disk.sdb1.out‚0( proc.memory.availableMB‚ 1299.8359375) disk./.usagePercent‚75.84599731607969 disk.sda1.out‚0 disk.nvme0n1.out‚ 41131.8125 disk.sdb.in‚ 0.92578125 disk.sda.in‚ 4.38671875 disk.nvme0n1.in‚ 603.58984375 disk.sdb.out‚856.85498046875! disk.sda2.out‚856.85498046875# disk.nvme1n1.out‚44761.91015625 network.recv‚ 2.421566e+06 disk.dm-0.in‚ 1102.390625 disk.sdb1.in‚0 disk.sdb2.in‚ 0.92578125$ memory_percent‚92.06643104553223* proc.memory.percent‚18.320131301879883 cpu‚25.212672875292007 proc.cpu.threads‚17 disk.md0.out‚ 825.6484375ɹŽ”†‚(:‰ ã»êÔ”À gpu.0.gpu‚32.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚170.891' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.72275 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0;Iû]‡‚(:Ò ã»êÔ”ÚŒ disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-0.out‚ 86018.703125 disk.md0p1.in‚5.3125 disk.sda.out‚858.6865234375 disk.sdb.out‚858.6865234375 disk.sda2.out‚858.6865234375 disk.md0.out‚ 827.421875 disk.sdb2.out‚858.6865234375 disk.dm-0.in‚ 1102.390625 disk.sda.in‚ 4.38671875$ memory_percent‚92.06798076629639 disk.rbd0.out‚2.1875 disk.nvme1n1.in‚ 498.80078125 disk.sdb1.in‚0 disk.md0p1.out‚ 827.421875# disk.nvme0n1.out‚41203.99609375) disk./.usagePercent‚75.84948532880115 disk.md0.in‚5.3125 disk.rbd2.in‚0) proc.memory.availableMB‚ 1299.58203125 disk.sda2.in‚ 4.38671875 disk.nvme0n1.in‚ 603.58984375 cpu‚25.212308634551206# disk.nvme1n1.out‚44814.70703125 disk.sdb1.out‚0 disk.rbd2.out‚2.125 disk.sdb.in‚ 0.92578125 network.sent‚ 1.63322e+07% disk./.usageGB‚2712.7449913024902 disk.sdb2.in‚ 0.92578125" proc.memory.rssMB‚ 3001.5703125 proc.cpu.threads‚17 network.recv‚ 2.432336e+06 disk.rbd0.in‚0* proc.memory.percent‚18.320131301879883[Klµ2ˆ- _wandbruntime‚7495  _runtime‚7495˜M’9ƉÀ & lr-AdamW/pg2‚0.0004536621171263219 trainer/global_step‚6449 $ _timestamp‚1.788517870239991e+09 _runtime‚7495.710330664 _step‚527 & lr-AdamW/pg1‚0.0004536621171263219ì½ýjËŠÅ & lr-AdamW/pg1‚0.0004536621171263219 & lr-AdamW/pg2‚0.0004536621171263219 trainer/global_step‚6449 $ _timestamp‚1.788517870239991e+09 _runtime‚7495.710330664 _step‚527/<õÿ2‹- _wandbruntime‚7495  _runtime‚74951rw…ü'ŒöO UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407872 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 161807566 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚518670762545040 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 _step‚528 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚537848826891776 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 256994304 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955591168 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955591168 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚537848826891776 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 121516416 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚593595880633344 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 161833654 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7655878490624 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450780 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26088 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 121515800 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 233054208 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 294998350 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚82804855 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917410816 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410816 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚601251992178176 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚601251735183872 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7656111544832 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 133164694 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 161807566 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7333964278784 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 161833654 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚530515780023808 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1187 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚525976473300892 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411420 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7333964278784 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚614 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚593595856693248 trainer/global_step‚6449 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚537849744302592 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7333064870912 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407872 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26090 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 _runtime‚7495.711490509 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 294972260 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚530515762020864 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7305727795212 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1212 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 % _timestamp‚1.7885178702410595e+09 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7333064870912 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚82804853 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007241 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26088 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177047040 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚537849744302592 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚616 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚525977387751672 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7306625206632 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 133164694 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚518670745505680 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411420 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7333964278784 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚614 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚593595856693248 trainer/global_step‚6449 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚537849744302592 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7333064870912 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407872 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26090 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 _runtime‚7495.711490509 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 294972260 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚530515762020864 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7305727795212 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1212 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 % _timestamp‚1.7885178702410595e+09 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7333064870912 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚82804853 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007241 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26088 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177047040 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚537849744302592 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚616 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚525977387751672 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7306625206632 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 133164694 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚518670745505680 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚537848826891776 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 256994304 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955591168 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955591168 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚537848826891776 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 121516416 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚593595880633344 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 161833654 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7655878490624 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450780 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26088 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 121515800 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 233054208 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 294998350 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚82804855 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917410816 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410816 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚601251992178176 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚601251735183872 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7656111544832 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 133164694 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 161807566 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7333964278784 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 161833654 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚530515780023808ùþ2Ž- _wandbruntime‚7496  _runtime‚7496‡µ÷ '€N REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7307733868158 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7657044557312 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚526056476507326 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚530595392316416 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411628 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 295016749 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 121534420 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450988 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7657267112448 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚537930487414784 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚537930487414784 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7306836456530 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚598 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚537929569990656 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7335077095424 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26114 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 161858271 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚530595410319360 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚596 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 133184594 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚593685175913984 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 trainer/global_step‚6449 _runtime‚7496.888529492 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177047040 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚82817229 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 133184592 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 295042865 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 133184592 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007241 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917424128 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚518748742639168 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899421184 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 161858271 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 121533822 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 246495232 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚601342220471296 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚601342466966528 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 222555136 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 295042865 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26114 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚38716593 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1187 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 _step‚529 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7885178714170873e+09 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7335077095424 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955591168 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899421184 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 133184594 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚530595410319360 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 398533632}Ñ‹ '…N OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 133184594 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚530595410319360 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 398533632 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7307733868158 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7657044557312 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚526056476507326 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚530595392316416 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411628 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 295016749 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 121534420 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450988 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7657267112448 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚537930487414784 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚537930487414784 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7306836456530 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚598 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚537929569990656 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7335077095424 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26114 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 161858271 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚530595410319360 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚596 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 133184594 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚593685175913984 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 trainer/global_step‚6449 _runtime‚7496.888529492 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177047040 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚82817229 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 133184592 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 295042865 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 133184592 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007241 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917424128 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚518748742639168 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899421184 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 161858271 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 121533822 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 246495232 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚601342220471296 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚601342466966528 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 222555136 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 295042865 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26114 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚38716593 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1187 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 _step‚529 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7885178714170873e+09 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7335077095424 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955591168 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899421184‘[ÉÌ52‘- _wandbruntime‚7496  _runtime‚7496Yèñ [’Õ .loss/FBt4/detection_loss‚2.392258948749966 _runtime‚7496.890171379 1loss/FBt1/assignment_loss‚0.07658365574316121 /loss/FBt2/detection_loss‚0.7815191874492772 1loss/FBt2/assignment_loss‚0.10476313749250744 1loss/FBt3/assignment_loss‚0.15247436273299778 0loss/SRqqt2/detection_loss‚1.093744708225131 2loss/SRqqt4/assignment_loss‚2.0030949595051686 .loss/FRt1/detection_loss‚2.042435414481904 epoch‚5 0loss/FRt4/assignment_loss‚3.6752325392538503 2loss/SRqqt1/assignment_loss‚1.7837810054262901 2loss/SRqqt2/assignment_loss‚0.7408087969857401 /loss/FRt3/detection_loss‚2.2516930943658187 .loss/FRt4/detection_loss‚2.864940749060723 % _timestamp‚1.7885178714200118e+09 0loss/SRqqt1/detection_loss‚1.721576067836995 &loss/total_loss‚40.333278006776325 trainer/global_step‚6449 0loss/FRt2/assignment_loss‚1.5398945132127175 2loss/SRqqt3/assignment_loss‚1.0893945406692533 0loss/SRqqt3/detection_loss‚1.513232233719184 /loss/FBt1/detection_loss‚3.0745044589042663 /loss/FBt3/detection_loss‚2.3421152333418527 _step‚530 0loss/FRt3/assignment_loss‚1.9514540891314662 0loss/FBt4/assignment_loss‚0.2214530722151393 /loss/FRt2/detection_loss‚1.5313547061971173 0loss/SRqqt4/detection_loss‚2.642491049000195 /loss/FRt1/assignment_loss‚2.7424774830755983lßa`“Ú 0loss/SRqqt3/detection_loss‚1.513232233719184 /loss/FBt1/detection_loss‚3.0745044589042663 /loss/FBt3/detection_loss‚2.3421152333418527 _step‚530 0loss/FRt3/assignment_loss‚1.9514540891314662 0loss/FBt4/assignment_loss‚0.2214530722151393 /loss/FRt2/detection_loss‚1.5313547061971173 0loss/SRqqt4/detection_loss‚2.642491049000195 /loss/FRt1/assignment_loss‚2.742477483075598 .loss/FBt4/detection_loss‚2.392258948749966 _runtime‚7496.890171379 1loss/FBt1/assignment_loss‚0.07658365574316121 /loss/FBt2/detection_loss‚0.7815191874492772 1loss/FBt2/assignment_loss‚0.10476313749250744 1loss/FBt3/assignment_loss‚0.15247436273299778 0loss/SRqqt2/detection_loss‚1.093744708225131 2loss/SRqqt4/assignment_loss‚2.0030949595051686 .loss/FRt1/detection_loss‚2.042435414481904 epoch‚5 0loss/FRt4/assignment_loss‚3.6752325392538503 2loss/SRqqt1/assignment_loss‚1.7837810054262901 2loss/SRqqt2/assignment_loss‚0.7408087969857401 /loss/FRt3/detection_loss‚2.2516930943658187 .loss/FRt4/detection_loss‚2.864940749060723 % _timestamp‚1.7885178714200118e+09 0loss/SRqqt1/detection_loss‚1.721576067836995 &loss/total_loss‚40.333278006776325 trainer/global_step‚6449 0loss/FRt2/assignment_loss‚1.5398945132127175 2loss/SRqqt3/assignment_loss‚1.0893945406692533’uI8A“”‚(:ˆ ò»êÔäíÔ€ gpu.0.gpu‚16.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚163.422' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚40.8555 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Òr~T•‚(:É ò»êÔü disk.sdb.in‚ 0.92578125 disk.md0.in‚5.3125 disk.sda1.out‚0 disk.md0p1.out‚ 828.5390625 disk.sdb.out‚ 859.83984375 disk.dm-0.out‚ 86195.59375 disk.nvme1n1.in‚ 498.80078125 disk.nvme0n1.in‚ 603.58984375" proc.memory.rssMB‚ 3001.5703125 network.sent‚ 1.635798e+07 disk.sda.in‚ 4.38671875 disk.sda2.out‚ 859.83984375" disk.nvme1n1.out‚ 44919.7109375 disk.md0.out‚ 828.5390625 disk.sdb2.in‚ 0.92578125 disk.sda1.in‚0) proc.memory.availableMB‚ 1299.06640625 network.recv‚ 2.450468e+06 disk.sda.out‚ 859.83984375 disk.sdb1.out‚0 disk.rbd2.in‚0 disk.dm-0.in‚ 1102.390625 disk.md0p1.in‚5.3125* proc.memory.percent‚18.320131301879883 disk.rbd0.in‚0$ memory_percent‚92.07112789154053 cpu‚25.21191197323502 proc.cpu.threads‚17) disk./.usagePercent‚75.85998989796768% disk./.usageGB‚2713.1206855773926 disk.rbd0.out‚2.1875 disk.rbd2.out‚2.125 disk.sdb2.out‚ 859.83984375" disk.nvme0n1.out‚ 41275.8828125 disk.sda2.in‚ 4.38671875 disk.sdb1.in‚0kC“–‚(:ˆ ¼êÔöÐèÿ gpu.0.gpu‚33.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚163.422' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚40.8555 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0bH(ƒX—‚(:Í ¼êÔ²ëŽ disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-0.out‚ 86372.8515625 proc.cpu.threads‚17 network.sent‚ 1.6452179e+07 disk.sda.in‚ 4.38671875 disk.rbd0.in‚0 disk.sdb1.out‚0 disk.rbd2.out‚2.125 disk.sda.out‚ 861.77734375 disk.sda2.in‚ 4.38671875 disk.rbd0.out‚2.1875 disk.rbd2.in‚0 disk.md0p1.out‚ 830.390625* proc.memory.percent‚18.320131301879883) disk./.usagePercent‚75.86695312414672% disk./.usageGB‚2713.3697242736816 disk.md0.in‚5.3125 disk.dm-0.in‚ 1102.390625 disk.md0p1.in‚5.3125" proc.memory.rssMB‚ 3001.5703125 cpu‚25.211530259556955$ memory_percent‚92.06817150115967 disk.sdb.in‚ 0.92578125) proc.memory.availableMB‚ 1299.55078125 network.recv‚ 2.454804e+06 disk.sda2.out‚ 861.77734375 disk.sdb1.in‚0 disk.sdb2.in‚ 0.92578125 disk.nvme0n1.in‚ 603.58984375# disk.nvme1n1.out‚45024.91015625 disk.sdb2.out‚ 861.77734375# disk.nvme0n1.out‚41347.94140625 disk.sdb.out‚ 861.77734375 disk.nvme1n1.in‚ 498.80078125 disk.md0.out‚ 830.390625 ˆ”˜‚(:‰ ¼êÔŠéëÿ gpu.0.gpu‚37.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚169.333' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.33325 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0YQ·f™‚(:Û ¼êÔÅò„€ disk.dm-0.out‚86373.07421875 disk.sdb.in‚ 0.92578125 disk.sda.in‚ 4.38671875 disk.rbd0.out‚2.1875# disk.nvme1n1.out‚45025.12890625 disk.md0p1.in‚5.3125 disk.md0p1.out‚ 831.77734375 network.sent‚ 1.6460385e+07) disk./.usagePercent‚75.86695152423873 disk.sdb1.in‚0 disk.sda1.in‚0 disk.sda2.in‚ 4.38671875 disk.nvme1n1.in‚ 498.80078125 cpu‚25.21115075212243 network.recv‚ 2.459599e+06 disk.nvme0n1.in‚ 603.58984375 disk.sda.out‚863.20556640625# memory_percent‚92.0670747756958 proc.cpu.threads‚17 disk.dm-0.in‚ 1102.390625" disk.nvme0n1.out‚ 41347.9453125 disk.sdb.out‚863.20556640625 disk.md0.in‚5.3125 disk.rbd2.in‚0" proc.memory.rssMB‚ 3001.5703125% disk./.usageGB‚2713.3696670532227! disk.sdb2.out‚863.20556640625 disk.sda1.out‚0* proc.memory.percent‚18.320131301879883! disk.sda2.out‚863.20556640625 disk.rbd0.in‚0 disk.sdb2.in‚ 0.92578125 disk.sdb1.out‚0 disk.md0.out‚ 831.77734375) proc.memory.availableMB‚ 1299.73046875 disk.rbd2.out‚2.125!ópË’š‚(:‡ Ÿ¼êÔ¨ŽÒý gpu.0.gpu‚56.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚36.0 gpu.0.powerWatts‚114.408' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.602 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0áY ù]›‚(:Ò Ÿ¼êÔšÖìÿ# disk.nvme1n1.out‚45234.97265625 disk.dm-0.in‚ 1102.390625% disk./.usageGB‚2713.3678131103516 disk.nvme0n1.in‚ 603.58984375 disk.sda.in‚ 4.38671875 disk.sdb1.in‚0* proc.memory.percent‚18.320131301879883# disk.nvme0n1.out‚41432.03515625 disk.sda2.in‚ 4.38671875 disk.dm-0.out‚ 86667.0078125 network.recv‚ 2.463908e+06 disk.md0p1.out‚ 833.984375$ memory_percent‚92.06762313842773 cpu‚25.21070067121126 disk.md0.out‚ 833.984375 disk.sdb.out‚865.4775390625 disk.nvme1n1.in‚ 498.80078125" proc.memory.rssMB‚ 3001.5703125 proc.cpu.threads‚17 network.sent‚ 1.6466182e+07 disk.sda1.in‚0 disk.sda2.out‚865.4775390625 disk.rbd0.in‚0 disk.rbd0.out‚2.1875 disk.rbd2.out‚2.125 disk.sdb2.out‚865.4775390625 disk.md0p1.in‚5.3125 disk.sdb.in‚ 0.92578125 disk.sdb1.out‚0' proc.memory.availableMB‚ 1299.640625) disk./.usagePercent‚75.86689968722013 disk.md0.in‚5.3125 disk.rbd2.in‚0 disk.sda1.out‚0 disk.sda.out‚865.4775390625 disk.sdb2.in‚ 0.92578125Ÿmtð2œ- _wandbruntime‚7555  _runtime‚7555ÚÙwÉà ' lr-AdamW/pg1‚0.00044747917413606297 ' lr-AdamW/pg2‚0.00044747917413606297 trainer/global_step‚6499 % _timestamp‚1.7885179304151106e+09 _runtime‚7555.885548428 _step‚531-p‡ØÎžÈ _runtime‚7555.885548428 _step‚531 ' lr-AdamW/pg1‚0.00044747917413606297 ' lr-AdamW/pg2‚0.00044747917413606297 trainer/global_step‚6499 % _timestamp‚1.7885179304151106e+09“ëíº2Ÿ- _wandbruntime‚7555  _runtime‚7555'.Ç*ó öO D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 % _timestamp‚1.7885179304162242e+09 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚522569767251440 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177047040 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚534497276798464 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚605775794977792 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1212 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytesÑc;÷ ".small_pool.current‚ 230957056 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 163066556 trainer/global_step‚6499 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 297200062 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 134159594 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚541885996249600 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚541886913660416 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 163066556 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7361174452320 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚541885996249600 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚83423653 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚38995375 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7714208204800 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚589 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚522569750212080 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7388719451136 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 163040468 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚587 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007241 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 230957056 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 163066556 trainer/global_step‚6499 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 297200062 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 134159594 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚541885996249600 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚541886913660416 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 163066556 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7361174452320 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚541885996249600 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚83423653 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚38995375 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7714208204800 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚589 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚522569750212080 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7388719451136 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 163040468 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚587 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007241 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚522569767251440 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177047040”›Ð’2¢- _wandbruntime‚7557  _runtime‚7557DÎGã'£€N F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚83436029 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899421184 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚605866526588416 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177047040 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚522647747345568 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7389832254464 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 297244551 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 163091173 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 398533632 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7715365697536 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 122437722 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚39001085 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955591168 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 297270667 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26114 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 _runtime‚7557.105153101 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899421184 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411628 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 122437114 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 222555136 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚605866280093184 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 297244551 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984Bh𛤅N [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚534576925096960 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚541967656772608 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 163065059 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31686 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 297270667 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚39001691 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚598151160890880 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26114 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚541966739348480 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 297270667 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26114 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 _runtime‚7557.105153101 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899421184 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411628 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 122437114 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 222555136 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚605866280093184 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 297244551 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚83436029 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899421184 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚605866526588416 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177047040 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚522647747345568 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7389832254464 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 297244551 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 163091173 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 398533632 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7715365697536 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 122437722 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚39001085 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955591168 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 PDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917410816 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚545923165607424 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚538478791576064 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚538478791576064 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚600 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚84042455 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955591168 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407872 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7444374031360 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 123324096 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 242314240 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 135154494 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26088 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7445273439232 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1212 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚526468771957840 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7445273439232 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177047040 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚598 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7772198172160 trainer/global_step‚6549 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410816 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007241 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 398533632 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 135154496 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 164273370 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450780 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214665 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7417518520848 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚610300000879104 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 _runtime‚7617.144152255 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1187 _step‚536 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚545924083018240 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7444374031360 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚545923165607424 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 135154494 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚39281641 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚610299758564864 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 135154496 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 218374144 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚545923165607424 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 135154494 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚39281641 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚610299758564864 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 135154496 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 218374144 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917410816 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚545923165607424 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚538478791576064 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚538478791576064 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚600 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚84042455 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955591168 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407872 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7444374031360 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 123324096 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 242314240 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 GADeviceStatsMonitor.OÎx=™ on_train_batch_start/oversize_segments.current‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 135154494 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26088 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7445273439232 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1212 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚526468771957840 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7445273439232 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177047040 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚598 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7772198172160 trainer/global_step‚6549 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410816 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007241 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 398533632 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 135154496 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 164273370 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450780 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214665 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7417518520848 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚610300000879104 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416˜ëNû2¶- _wandbruntime‚7618  _runtime‚7618û4¾'·€N I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26114 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 135174394 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚39286646 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1187 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 135174392 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚587 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 236009472 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917424128 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚610390493614080 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 299472353 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚84054831 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 164324075 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7773159617024 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7445486834688 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 398533632 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7446386255872 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚84054829 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7418627182374 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚538558421871616 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 299498469 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 164297961 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7773371686400 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177047040 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31686 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚602617121927680 _runtime‚7618.352318527 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚546004826130432 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚538558421871616 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 123341475 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 212069376 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 135174394 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899421184 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚610390257604608 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 trainer/global_step‚6549 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 135174392 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 164324075 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚546003908706304 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7445486834688 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _step‚537 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26114 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚533964464783354 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411628 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007241 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899421184 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚546004826130432h¼- '¸…N I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚39287233 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 % _timestamp‚1.7885179928802328e+09 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7417729770746 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899421184 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚546004826130432 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26114 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 135174394 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚39286646 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1187 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 135174392 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚587 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 236009472 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917424128 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚610390493614080 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 299472353 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚84054831 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 164324075 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7773159617024 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7445486834688 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 398533632 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7446386255872 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚84054829 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7418627182374 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚538558421871616 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 299498469 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 164297961 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7773371686400 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177047040 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31686 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚602617121927680 _runtime‚7618.352318527 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚546004826130432 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚538558421871616 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 123341475 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 212069376 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 135174394 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899421184 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚610390257604608 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 trainer/global_step‚6549 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 135174392 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 164324075 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚546003908706304 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7445486834688 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _step‚537 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26114 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚533964464783354 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411628 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007241 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 FDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450780 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚549960334965248 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411420 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26088 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 398533632 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 136149394 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007241 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177047040 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚614823969983488 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 _runtime‚7674.153286878 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 252800000 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1212 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚542460306353664 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚542460306353664 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7500928019456 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26088 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 124227624 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 136149396 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7472067766536 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7830206239744 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407872 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚549961252376064 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 228859904 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚537840741842196 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚530367759624880 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚549961252376064 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚549960334965248 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7500028611584 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚39567014 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚643 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955591168 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 _step‚540 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 165532360 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚84661255 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 _step‚540 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 165532360 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚84661255 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450780 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚549960334965248 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411420 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26088 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 398533632 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 136149394 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007241 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177047040 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚614823969983488 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 _runtime‚7674.153286878 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 252800000 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1212 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚542460306353664 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚542460306353664 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7500928019456 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26088 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 124227624œÛTY2Ê- _wandbruntime‚7675  _runtime‚7675\™4E'ËÿM UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7831123778560 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _step‚541 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 136169292 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚614914182802944 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚607083059024384 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 165530863 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 301726271 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 136169294 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 301700155 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 136169294 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 124245754 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899421184 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚39572747 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899421184 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26114 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚624 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7474073839482 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7831358916608 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 259078144 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 165556977 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 301700155 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 165530863 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 398533632 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26114 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7502040836096 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 235138048 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 $ _timestamp‚1.788518049707809e+09 _runtime‚7675.180042035 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚542539954652160 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚614914441881088 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚537918916146862 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411628 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007241 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917424128 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚542539954652160 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955591168 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚542539936649216 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7501141414912 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 124246378 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚530445739719008 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7473176427854 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚550041995488256 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 136169292 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31686 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0¾-ñxÌ„N L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 124245754 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899421184 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚39572747 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899421184 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26114 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚624 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7474073839482 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7831358916608 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 259078144 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 165556977 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 301700155 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 165530863 QCDevicŠœ²:’eStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 398533632 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26114 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7502040836096 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 235138048 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 $ _timestamp‚1.788518049707809e+09 _runtime‚7675.180042035 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚542539954652160 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚614914441881088 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚537918916146862 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411628 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007241 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917424128 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚542539954652160 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955591168 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚542539936649216 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7501141414912 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 124246378 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚530445739719008 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7473176427854 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚550041995488256 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 136169292 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31686 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7831123778560 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _step‚541 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 136169292 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚614914182802944 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚607083059024384 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 165530863 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 301726271 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 136169294 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 301700155 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 136169294 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 FDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚553997504323072 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚619347609581568 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 137144294 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 166739174 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 398533632 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26088 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚85280055 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 224665600 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 125132633 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7528411835064 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7527514423644 _step‚544 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚546441839134208 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177047040 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚546441821131264 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 166739174 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚546441821131264 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1212 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214665 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 trainer/global_step‚6649 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407872 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 137144296 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955591168 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚39852578 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26090 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚541794278754924 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 166765262 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410816 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚546441839134208 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 303909558 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚39851954 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1187 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7556582599680 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚534266781370640 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 _runtime‚7727.175041375 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚553998421733888 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917410816 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7556582599680 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7887908741120 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 137144294 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564Mƒ/ÝûO E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚619347858187264 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 248605696 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚541794278754924 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 166765262 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410816 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚546441839134208 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 303909558 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚39851954 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1187 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7556582599680 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚534266781370640 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 _runtime‚7727.175041375 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚553998421733888 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917410816 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7556582599680 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7887908741120 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 137144294 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7555683191808 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚553997504323072 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚85280053 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7888133406720 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007241 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚553997504323072 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚619347609581568 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 137144294 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 166739174 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 398533632 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26088 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚85280055 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 224665600 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 125132633 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7528411835064 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7527514423644 _step‚544 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚546441839134208 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177047040 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚546441821131264 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 166739174 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚546441821131264 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1212 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214665 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 trainer/global_step‚6649 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407872 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 137144296 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955591168 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚39852578 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26090 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 303954073 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7557695416320 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚541874281961358 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 398533632 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 trainer/global_step‚6649 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7556795995136 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 137164194 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 166789879 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚619438133627392 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1187 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899421184 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚534344761464768 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚39857782 % _timestamp‚1.7885181026788893e+09 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 125150211 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚39858382 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26114 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7556795995136 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 125150813 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚611549020061184 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚534344744425408 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007241 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 303954073 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7557695416320 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚541874281961358 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 398533632 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 trainer/global_step‚6649 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7556795995136 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 137164194 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 166789879 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚619438133627392 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1187 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899421184 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚534344761464768 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚39857782 % _timestamp‚1.7885181026788893e+09 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 125150211 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚39858382 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26114 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7556795995136 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 125150813 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚611549020061184 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚534344744425408 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007241 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚558034673680896 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007241 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 306111270 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 126037158 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚538165769037680 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 % _timestamp‚1.7885181569552772e+09 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 167972076 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚550423335908864 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955591168 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7612237179904 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7611337772032 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 233054208 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7946470205440 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚623871899028480 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 167998164 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7583858492172 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 167998164 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚545748730118432 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚558035591091712 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚623 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚550423353911808 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410816 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 306137360 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚615925685817344 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 256994304 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 138139196 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26088 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917410816 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚625 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 126036533 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 trainer/global_step‚6699 _runtime‚7782.425706736 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177047040 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚558034673680896 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚615925661877248 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1212 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411420 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7582961080752 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 138139196 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚550423335908864 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚558034673680896 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007241 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 306111270 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 126037158 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚538165769037680 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 % _timestamp‚1.7885181569552772e+09 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 167972076 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚550423335908864 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955591168 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7612237179904 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7611337772032 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 233054208 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7946470205440 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚623871899028480 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 167998164 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7583858492172 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 167998164 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚545748730118432 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚558035591091712 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚623 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚550423353911808 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410816 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 306137360 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚615925685817344 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 256994304 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 138139196 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26088 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917410816 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚625 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 126036533 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 trainer/global_step‚6699 _runtime‚7782.425706736 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177047040 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚558034673680896 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚615925661877248 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1212 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411420 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7582961080752 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 138139196 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚550423335908864 JDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚550502966204416 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1187 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177047040 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31686 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 226749440 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177047040 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 168022781 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 126054680 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚40143451 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 306155759 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚616015005038080 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚545828733324866 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7612450575360 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450988 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚623962629874176 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚550502984207360 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚40144065 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚550502966204416 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 138159094 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26114 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚623962379184640 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚538243749131808 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 167996667 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚85911231 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955591168 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚558116334203904 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 138159094 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚550502984207360 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 138159092 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899421184 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899421184 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 138159092 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917424128 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7947398086656 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 % _timestamp‚1.7885181579597101e+09 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7612450575360 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚616 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚614 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 250689536 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7947624836096 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 168022781 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 306181875 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚616014981097984 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7584069742070 trainer/global_step‚6699Ž·Àðjò„N RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚550502984207360 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚40144065 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚550502966204416 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 138159094 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26114 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚623962379184640 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚538243749131808 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 167996667 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚85911231 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955591168 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚558116334203904 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 138159094 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚550502984207360 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 138159092 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899421184 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899421184 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 138159092 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917424128 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7947398086656 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 % _timestamp‚1.7885181579597101e+09 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7612450575360 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚616 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚614 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 250689536 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7947624836096 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 168022781 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 306181875 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚616014981097984 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7584069742070 trainer/global_step‚6699 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 306155759 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚85911229 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚550502966204416 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1187 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177047040 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31686 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 226749440 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177047040 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 168022781 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 126054680 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚40143451 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 306155759 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚616015005038080 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚545828733324866 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7612450575360 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450988 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚623962629874176 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008¥æÑpó2ó- _wandbruntime‚7783  _runtime‚7783i i\ôÖ 2loss/SRqqt3/assignment_loss‚1.2920445358021218 .loss/FBt4/detection_loss‚2.484348102049394 2loss/SRqqt1/assignment_loss‚1.1425754623487592 /loss/FRt1/detection_loss‚1.9695585777349605 /loss/FRt2/detection_loss‚1.4165327986081442 _runtime‚7783.432442227 .loss/FBt1/detection_loss‚2.003330091635386 epoch‚5 /loss/FRt2/assignment_loss‚1.579481840133667 1loss/FBt3/assignment_loss‚0.41950939447893426 1loss/FBt4/assignment_loss‚0.36236933675867145 1loss/FBt2/assignment_loss‚0.23384019740212333 /loss/FRt3/detection_loss‚2.5332977842237496 /loss/FBt2/detection_loss‚0.7081942899007495 %loss/total_loss‚40.07065535251195 .loss/FRt4/detection_loss‚3.082100944519043 1loss/SRqqt1/detection_loss‚2.0628490146053466 0loss/SRqqt4/detection_loss‚2.629806766143212 % _timestamp‚1.7885181579623117e+09 0loss/FRt1/assignment_loss‚2.5350862312316895 0loss/FRt4/assignment_loss‚3.7096480178833007 2loss/SRqqt2/assignment_loss‚0.9527794754105634 1loss/FBt1/assignment_loss‚0.32337331656987467 1loss/SRqqt3/detection_loss‚1.3715469845031436 _step‚550 /loss/FRt3/assignment_loss‚2.465972806956317 2loss/SRqqt4/assignment_loss‚1.5299330335110426 0loss/SRqqt2/detection_loss‚1.093159172737944 /loss/FBt3/detection_loss‚2.1693171773638045 trainer/global_step‚6699»A^øaõÛ 2loss/SRqqt3/assignment_loss‚1.2920445358021218 .loss/FBt4/detection_loss‚2.484348102049394 2loss/SRqqt1/assignment_loss‚1.1425754623487592 /loss/FRt1/detection_loss‚1.9695585777349605 /loss/FRt2/detection_loss‚1.4165327986081442 _runtime‚7783.432442227 .loss/FBt1/detection_loss‚2.003330091635386 epoch‚5 /loss/FRt2/assignment_loss‚1.579481840133667 1loss/FBt3/assignment_loss‚0.41950939447893426 1loss/FBt4/assignment_loss‚0.36236933675867145 1loss/FBt2/assignment_loss‚0.23384019740212333 /loss/FRt3/detection_loss‚2.5332977842237496 /loss/FBt2/detection_loss‚0.7081942899007495 %loss/total_loss‚40.07065535251195 .loss/FRt4/detection_loss‚3.082100944519043 1loss/SRqqt1/detection_loss‚2.0628490146053466 0loss/SRqqt4/detection_loss‚2.629806766143212 % _timestamp‚1.7885181579623117e+09 0loss/FRt1/assignment_loss‚2.5350862312316895 0loss/FRt4/assignment_loss‚3.7096480178833007 2loss/SRqqt2/assignment_loss‚0.9527794754105634 1loss/FBt1/assignment_loss‚0.32337331656987467 1loss/SRqqt3/detection_loss‚1.3715469845031436 _step‚550 /loss/FRt3/assignment_loss‚2.465972806956317 2loss/SRqqt4/assignment_loss‚1.5299330335110426 0loss/SRqqt2/detection_loss‚1.093159172737944 /loss/FBt3/detection_loss‚2.1693171773638045 trainer/global_step‚6699¦‚9tb“ö‚(:ˆ ¾êÔ¦Ìåþ gpu.0.gpu‚52.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚134.298' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚33.5745 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0zý `÷‚(:Õ ¾êÔŽù”ÿ network.sent‚ 1.6964559e+07) disk./.usagePercent‚75.85282657663038 disk.rbd0.in‚0 disk.nvme1n1.out‚ 46290.03125$ memory_percent‚92.07010269165039 disk.rbd2.out‚2.125 disk.md0p1.out‚ 859.8046875 disk.nvme0n1.in‚ 603.58984375 disk.md0.out‚ 859.8046875 disk.sdb2.in‚ 0.92578125* proc.memory.percent‚18.320131301879883 disk.rbd2.in‚0$ disk./.usageGB‚2712.864490509033 disk.sdb.out‚892.22509765625 disk.rbd0.out‚2.1875! disk.sdb2.out‚892.22509765625 cpu‚25.204774998579342 disk.sda1.out‚0 disk.dm-0.in‚ 1102.390625# disk.nvme0n1.out‚42320.98046875 disk.sdb1.in‚0" proc.memory.rssMB‚ 3001.5703125 disk.sda1.in‚0 disk.sdb.in‚ 0.92578125 proc.cpu.threads‚17 disk.dm-0.out‚88611.01171875 disk.md0p1.in‚5.3125 disk.sda.out‚892.22509765625 disk.sda2.in‚ 4.38671875! disk.sda2.out‚892.22509765625 disk.nvme1n1.in‚ 498.80078125 disk.sdb1.out‚0 network.recv‚ 2.53585e+06 disk.sda.in‚ 4.38671875 disk.md0.in‚5.3125' proc.memory.availableMB‚ 1299.234375+¡µ¨”ø‚(:‰ ž¾êÔ³èòÿ gpu.0.gpu‚56.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚37.0 gpu.0.powerWatts‚145.771' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚36.44275 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ø£Í{Tù‚(:É ž¾êÔ™–£€ disk.md0.in‚5.3125 disk.md0.out‚ 861.15234375( proc.memory.availableMB‚ 1298.8515625 cpu‚25.204438868544408 proc.cpu.threads‚17 disk.sda2.out‚ 893.6328125 disk.sda1.in‚0 disk.md0p1.out‚ 861.15234375 disk.sdb.out‚ 893.6328125 disk.sda2.in‚ 4.38671875 disk.nvme1n1.in‚ 498.80078125 disk.rbd2.in‚0 disk.rbd2.out‚2.125 disk.sdb.in‚ 0.92578125# disk.nvme1n1.out‚46366.51171875$ disk./.usageGB‚2712.868507385254 disk.dm-0.in‚ 1102.390625 disk.dm-0.out‚88795.33984375 disk.rbd0.in‚0 disk.rbd0.out‚2.1875 disk.sdb1.out‚0$ memory_percent‚92.07243919372559" proc.memory.rssMB‚ 3001.5703125 disk.sda1.out‚0 disk.nvme0n1.in‚ 603.58984375* proc.memory.percent‚18.320131301879883 disk.sdb1.in‚0 network.sent‚ 1.7072868e+07 network.recv‚ 2.542156e+06( disk./.usagePercent‚75.8529388901707 disk.sdb2.out‚ 893.6328125 disk.md0p1.in‚5.3125 disk.sda.in‚ 4.38671875 disk.sda.out‚ 893.6328125 disk.sdb2.in‚ 0.92578125! disk.nvme0n1.out‚ 42428.828125¥Ýc[ú‚(:Ð ­¾êÔèŸÜÿ disk.sdb1.out‚0 disk.md0.out‚ 862.70703125 disk.md0p1.out‚ 862.70703125 disk.rbd0.out‚2.1875 disk.rbd2.out‚2.125 disk.sda1.in‚0 network.sent‚ 1.707811e+07( disk./.usagePercent‚75.8529388901707 disk.sdb1.in‚0 disk.nvme0n1.in‚ 603.58984375) proc.memory.availableMB‚ 1299.23046875 cpu‚25.204115237839055 proc.cpu.threads‚17 disk.sda2.out‚ 895.251953125 disk.md0.in‚5.3125* proc.memory.percent‚18.320131301879883 network.recv‚ 2.545543e+06 disk.sda.in‚ 4.38671875 disk.nvme1n1.in‚ 498.80078125 disk.sda1.out‚0 disk.sdb.in‚ 0.92578125 disk.sdb.out‚ 895.251953125 disk.sda.out‚ 895.251953125! disk.nvme1n1.out‚ 46366.765625 disk.sdb2.in‚ 0.92578125# disk.nvme0n1.out‚42500.91015625 disk.sda2.in‚ 4.38671875 disk.sdb2.out‚ 895.251953125 disk.md0p1.in‚5.3125$ disk./.usageGB‚2712.868507385254 disk.rbd2.in‚0 disk.dm-0.in‚ 1102.390625 disk.dm-0.out‚88867.67578125# memory_percent‚92.0701265335083" proc.memory.rssMB‚ 3001.5703125 disk.rbd0.in‚0 úÚP’û‚(:‡ ­¾êÔ—ÅÑÿ gpu.0.gpu‚39.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚160.028' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚40.007 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0÷?ü¬Yü‚(:Î ¼¾êÔŒ†õþ proc.cpu.threads‚17 disk.sda1.in‚0 disk.nvme0n1.in‚ 603.58984375 disk.sda.out‚ 896.771484375 disk.rbd0.in‚0 disk.rbd2.in‚0) disk./.usagePercent‚75.85643063601078% disk./.usageGB‚2712.9933891296387 disk.md0p1.in‚5.3125 disk.sdb1.in‚0$ memory_percent‚92.06929206848145 disk.md0.in‚5.3125 disk.md0.out‚ 864.171875 network.sent‚ 1.7083216e+07 disk.dm-0.out‚88972.69140625" disk.nvme0n1.out‚ 42500.9140625 disk.sda.in‚ 4.38671875 disk.rbd0.out‚2.1875 cpu‚25.20376272137021 disk.sdb2.in‚ 0.92578125 disk.sda1.out‚0 disk.dm-0.in‚ 1102.390625 disk.sdb.out‚ 896.771484375 disk.nvme1n1.in‚ 498.80078125 disk.sda2.in‚ 4.38671875" proc.memory.rssMB‚ 3001.5703125 disk.md0p1.out‚ 864.171875 network.recv‚ 2.54938e+06 disk.sdb2.out‚ 896.771484375# disk.nvme1n1.out‚46471.77734375( proc.memory.availableMB‚ 1299.3671875 disk.sdb.in‚ 0.92578125 disk.sda2.out‚ 896.771484375 disk.sdb1.out‚0 disk.rbd2.out‚2.125* proc.memory.percent‚18.320131301879883ê6£I”ý‚(:‰ ¼¾êÔ¬£ç‹ gpu.0.gpu‚29.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚168.051' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.01275 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0b®2þ- _wandbruntime‚7837  _runtime‚7837IˆvÇÿÁ & lr-AdamW/pg1‚0.0004165320187381999 & lr-AdamW/pg2‚0.0004165320187381999 trainer/global_step‚6749 % _timestamp‚1.7885182122375088e+09 _runtime‚7837.708100797 _step‚551~6SŽÌ€Æ & lr-AdamW/pg1‚0.0004165320187381999 & lr-AdamW/pg2‚0.0004165320187381999 trainer/global_step‚6749 % _timestamp‚1.7885182122375088e+09 _runtime‚7837.708100797 _step‚551§Ï^ƒU2- _wandbruntime‚7837  _runtime‚7837ç¤ü'‚öO RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 126941302 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚86517655 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚554404868689408 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 256994304 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚542064773744080 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚40423647 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚562072760449536 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7666992352256 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 139134096 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7667891760128 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚620 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955591168 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 398533632 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚562072760449536 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 308339072 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955591168 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚618 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1187 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚628396330378240 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450780 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1212 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚40423029 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚620391646854144 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚549703181481940 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177047040 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚620391622914048 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8004450469888 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407872 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 308365162 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚562071843038720 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚554404850686464 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 _step‚552 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410816 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚562071843038720 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _runtime‚7837.709659784 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407872 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7666992352256 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 % _timestamp‚1.7885182122388785e+09ý3³*KƒûO WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚562072760449536 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 308339072 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955591168 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚618 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1187 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚628396330378240 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450780 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1212 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚40423029 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚620391646854144 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚549703181481940 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177047040 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚620391622914048 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8004450469888 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407872 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 308365162 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚562071843038720 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚554404850686464 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 _step‚552 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410816 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚562071843038720 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _runtime‚7837.709659784 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407872 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7666992352256 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 % _timestamp‚1.7885182122388785e+09 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 126941302 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚86517655 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚554404868689408 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 256994304 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚542064773744080 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚40423647 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚562072760449536 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7666992352256 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 139134096 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7667891760128 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚620 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955591168 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 398533632¨ãªf2„- _wandbruntime‚7838  _runtime‚7838Ž²Ôö'…€N K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 139153992 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚86530029 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚40428608 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 256980992 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚628486543311360 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚620480966074880 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚620480942134784 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31686 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7640413810806 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7669004576768 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚549782270237386 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177047040 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411628 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 308409677 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚562153503561728 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917424128 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 139153994 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 308409677 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 398533632 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚542142770877568 _runtime‚7838.671870716 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 126958637 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 169255683 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007241 _step‚553 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚554484498984960 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚554484480982016 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚604 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚40429210 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7639516399178 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 308383561 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 139153994 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 169255683 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 139153992 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 169229569 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚86530031 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 233040896 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 trainer/global_step‚6749 % _timestamp‚1.7885182132008836e+09 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚554484480982016 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899421184 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚86530031 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 233040896 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 trainer/global_step‚6749 % _timestamp‚1.7885182132008836e+09 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚554484480982016 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899421184 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚620480942134784 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31686 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7640413810806 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7669004576768 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚549782270237386 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177047040 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411628 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 308409677 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚562153503561728 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917424128 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 139153994 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 308409677 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 398533632 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚542142770877568 _runtime‚7838.671870716 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 126958637 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 169255683 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007241 _step‚553 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚554484498984960 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚554484480982016 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚604 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚40429210 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7639516399178 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 308383561 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 139153994 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 169255683 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 139153992 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 169229569 FDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚566109012396544 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚588 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917410816 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955591168 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚558386365464064 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚553657632845448 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411420 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚87136453 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7722646932480 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8062583801856 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177047040 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7723546340352 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 310566874 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚40710081 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8062808467456 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955591168 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214665 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 140128996 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7693854394968 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 170463968 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26088 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚632920167752704 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚624857607890944 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚545963778450480 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407872 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚624857583950848 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 trainer/global_step‚6799 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚586 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26088 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚545963795489840 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407872 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 140128994 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7722646932480 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 140128996 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7693854394968 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 170463968 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26088 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚632920167752704 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚624857607890944 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚545963778450480 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407872 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚624857583950848 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 trainer/global_step‚6799 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚586 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26088 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚545963795489840 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407872 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 140128994 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7722646932480 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚566109012396544 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚588 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917410816 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955591168 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚558386365464064 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚553657632845448 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411420 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚87136453 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7722646932480 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8062583801856 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177047040 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7723546340352 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 310566874 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0¬> ZØ2˜- _wandbruntime‚7897  _runtime‚7897À>Ü‚&™ÿM L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 127864597 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 244398080 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177047040 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚624946903171584 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚546041758544608 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚558466013762560 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚40716365 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚633010910041600 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚546041775583968 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26114 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚566190672919552 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7723759735808 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8063762471936 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 310637479 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 310637479 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚624946927111680 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 trainer/global_step‚6799 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7724659156992 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1187 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚633010665643520 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _runtime‚ 7897.37495994 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450988 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 140148894 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 170488585 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 170462471 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 127865196 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8063982929920 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 310611363 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 140148894 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚558466013762560 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 % _timestamp‚1.7885182719022386e+09 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411628 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899421184 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚558465995759616 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚558465995759616 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 398533632 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 310637479 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚624946927111680 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 trainer/global_step‚6799 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7724659156992 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1187 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚633010665643520 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _runtime‚ 7897.37495994 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450988 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 140148894 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 170488585 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 170462471 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 127865196 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8063982929920 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 310611363 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 140148894 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚558466013762560 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 % _timestamp‚1.7885182719022386e+09 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411628 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899421184 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚558465995759616 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚558465995759616 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 398533632 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 127864597 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 244398080 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177047040 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚624946903171584 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚546041758544608 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚558466013762560­JWÃ’2›- _wandbruntime‚7897  _runtime‚7897Óµ.NaœÛ 1loss/FBt2/assignment_loss‚0.16018123536239195 /loss/FRt2/detection_loss‚1.6358740124471334 1loss/SRqqt1/detection_loss‚1.7415472359416333 /loss/FBt2/detection_loss‚0.6234844653294064 .loss/FBt4/detection_loss‚3.755358949303627 /loss/FRt3/assignment_loss‚2.285338614881039 3loss/FBt4/assignment_loss‚0.0011828632796095917 /loss/FRt3/detection_loss‚2.3006683114916084 /loss/FBt3/detection_loss‚3.4713224702411227 _runtime‚7897.376142295 2loss/SRqqt4/assignment_loss‚1.6731647873635997 /loss/FRt4/detection_loss‚3.0769230008125303 1loss/SRqqt4/detection_loss‚2.8943933817473324 /loss/FRt1/assignment_loss‚2.304403841495514 2loss/SRqqt3/assignment_loss‚1.1195359241363465 1loss/SRqqt2/detection_loss‚0.9017472432209895 &loss/total_loss‚40.446513584883945 0loss/FRt4/assignment_loss‚2.7924864387512205 /loss/FRt1/detection_loss‚2.0635586442487632 .loss/FBt1/detection_loss‚2.641045000818041 epoch‚5 2loss/SRqqt1/assignment_loss‚0.9832718464684613 1loss/FBt1/assignment_loss‚0.06383136629948341 1loss/SRqqt3/detection_loss‚1.7514180880697334 trainer/global_step‚6799 /loss/FRt2/assignment_loss‚1.238578691774485 2loss/SRqqt2/assignment_loss‚0.8423916686899386 1loss/FBt3/assignment_loss‚0.12480550270993263 % _timestamp‚1.7885182719058526e+09 _step‚558ÆŠw"fà /loss/FRt1/assignment_loss‚2.304403841495514 2loss/SRqqt3/assignment_loss‚1.1195359241363465 1loss/SRqqt2/detection_loss‚0.9017472432209895 &loss/total_loss‚40.446513584883945 0loss/FRt4/assignment_loss‚2.7924864387512205 /loss/FRt1/detection_loss‚2.0635586442487632 .loss/FBt1/detection_loss‚2.641045000818041 epoch‚5 2loss/SRqqt1/assignment_loss‚0.9832718464684613 1loss/FBt1/assignment_loss‚0.06383136629948341 1loss/SRqqt3/detection_loss‚1.7514180880697334 trainer/global_step‚6799 /loss/FRt2/assignment_loss‚1.238578691774485 2loss/SRqqt2/assignment_loss‚0.8423916686899386 1loss/FBt3/assignment_loss‚0.12480550270993263 % _timestamp‚1.7885182719058526e+09 _step‚558 1loss/FBt2/assignment_loss‚0.16018123536239195 /loss/FRt2/detection_loss‚1.6358740124471334 1loss/SRqqt1/detection_loss‚1.7415472359416333 /loss/FBt2/detection_loss‚0.6234844653294064 .loss/FBt4/detection_loss‚3.755358949303627 /loss/FRt3/assignment_loss‚2.285338614881039 3loss/FBt4/assignment_loss‚0.0011828632796095917 /loss/FRt3/detection_loss‚2.3006683114916084 /loss/FBt3/detection_loss‚3.4713224702411227 _runtime‚7897.376142295 2loss/SRqqt4/assignment_loss‚1.6731647873635997 /loss/FRt4/detection_loss‚3.0769230008125303 1loss/SRqqt4/detection_loss‚2.8943933817473324®»)X [ž‚(:Ð ‡¿êÔŠíÿ disk.sda1.out‚0( proc.memory.availableMB‚ 1298.8359375 network.recv‚ 2.571748e+06 disk.sdb.out‚905.32861328125! disk.sda2.out‚905.32861328125 disk.md0.in‚5.3125 disk.dm-0.in‚ 1102.390625$ memory_percent‚92.07253456115723 proc.cpu.threads‚17$ disk./.usageGB‚2713.868095397949 disk.sdb.in‚ 0.92578125 disk.sdb1.in‚0* proc.memory.percent‚18.320131301879883 disk.nvme1n1.in‚ 498.80078125# disk.nvme1n1.out‚46996.97265625 disk.md0p1.in‚5.3125 cpu‚25.20224339061484 disk.sda.out‚905.32861328125 disk.sdb1.out‚0 disk.rbd2.in‚0 disk.rbd0.in‚0 disk.sdb2.in‚ 0.92578125 disk.nvme0n1.in‚ 603.58984375 disk.rbd0.out‚2.1875 disk.dm-0.out‚89857.61328125 disk.md0p1.out‚872.4375" proc.memory.rssMB‚ 3001.5703125! disk.nvme0n1.out‚ 42860.640625 disk.sda2.in‚ 4.38671875 disk.md0.out‚872.4375 disk.rbd2.out‚2.125 network.sent‚ 1.721433e+07) disk./.usagePercent‚75.88088778936576 disk.sda.in‚ 4.38671875! disk.sdb2.out‚905.32861328125 disk.sda1.in‚0. 5~”Ÿ‚(:‰ ‡¿êÔ•¬¢ÿ gpu.0.gpu‚26.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚164.989' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚41.24725 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ªq¶” ‚(:‰ –¿êÔª–þ gpu.0.gpu‚51.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚37.0 gpu.0.powerWatts‚118.751' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚29.68775 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0…1oÍ[¡‚(:Ð –¿êÔ—›†ÿ disk.md0p1.out‚ 873.9609375 disk.sdb.in‚ 0.92578125 network.recv‚ 2.578007e+06 disk.sda1.out‚0 disk.md0p1.in‚5.3125" disk.nvme0n1.out‚ 42932.5234375 disk.sdb.out‚906.90576171875 network.sent‚ 1.7322549e+07! disk.sda2.out‚906.90576171875 disk.sda1.in‚0 cpu‚25.201925772434$ disk./.usageGB‚2713.868095397949 disk.dm-0.in‚ 1102.390625$ memory_percent‚92.07415580749512) disk./.usagePercent‚75.88088778936576 disk.nvme1n1.in‚ 498.80078125! disk.sdb2.out‚906.90576171875( proc.memory.availableMB‚ 1298.5703125" proc.memory.rssMB‚ 3001.5703125 disk.sda.in‚ 4.38671875 disk.nvme1n1.out‚ 46997.125 disk.sdb2.in‚ 0.92578125* proc.memory.percent‚18.320131301879883 disk.md0.in‚5.3125 disk.md0.out‚ 873.9609375 disk.rbd2.in‚0 disk.nvme0n1.in‚ 603.58984375 disk.sda.out‚906.90576171875 disk.sda2.in‚ 4.38671875 disk.sdb1.in‚0 disk.rbd2.out‚2.125 disk.dm-0.out‚ 89929.6484375 proc.cpu.threads‚17 disk.rbd0.in‚0 disk.rbd0.out‚2.1875 disk.sdb1.out‚0)ÇQi”¢‚(:‰ ¥¿êÔÞ¼Úý gpu.0.gpu‚40.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚159.079' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚39.76975 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚07×p{K£‚(:À ¥¿êÔÔ¸Âÿ disk.nvme0n1.out‚43004.75% proc.memory.availableMB‚ 1298.8125* proc.memory.percent‚18.320131301879883# disk.nvme1n1.out‚46997.51953125 disk.md0.in‚5.3125 disk.sda1.in‚0 disk.md0p1.out‚876" proc.memory.rssMB‚ 3001.5703125 disk.sda2.in‚ 4.38671875! disk.sda2.out‚909.01318359375 disk.rbd0.out‚2.1875 disk.sdb1.in‚0$ memory_percent‚92.07267761230469! disk.sdb2.out‚909.01318359375 disk.sda1.out‚0 disk.rbd0.in‚0 disk.rbd2.in‚0 disk.rbd2.out‚2.125 disk.sdb2.in‚ 0.92578125 cpu‚25.20163640824102 disk.nvme1n1.in‚ 498.80078125 disk.nvme0n1.in‚ 603.58984375 disk.sdb1.out‚0 network.sent‚ 1.7327793e+07 disk.sda.in‚ 4.38671875 disk.sda.out‚909.01318359375 disk.sdb.in‚ 0.92578125 proc.cpu.threads‚17 network.recv‚ 2.581639e+06) disk./.usagePercent‚75.88089098918172$ disk./.usageGB‚2713.868209838867 disk.dm-0.in‚ 1102.390625 disk.md0p1.in‚5.3125 disk.sdb.out‚909.01318359375 disk.md0.out‚876 disk.dm-0.out‚90002.26953125=Yÿn’¤‚(:‡ ´¿êÔØóÿ gpu.0.gpu‚44.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚37.0 gpu.0.powerWatts‚163.09' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚40.7725 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Y¸‡a¥‚(:Ö ´¿êÔçü´€ network.recv‚ 2.586311e+06 disk.sda1.out‚0 disk.md0p1.in‚5.3125 disk.sdb.in‚ 0.92578125 disk.sda.out‚910.51220703125$ memory_percent‚92.07253456115723$ disk./.usageGB‚2713.868133544922 disk.sdb2.in‚ 0.92578125 disk.dm-0.out‚ 90264.6328125 disk.sdb.out‚910.51220703125 disk.rbd0.out‚2.1875 disk.dm-0.in‚ 1102.390625 disk.sda.in‚ 4.38671875! disk.sda2.out‚910.51220703125 disk.md0.in‚5.3125 disk.rbd2.out‚2.125 disk.md0p1.out‚ 877.44140625 cpu‚25.20131396906194" disk.nvme0n1.out‚ 43004.8515625 disk.sda2.in‚ 4.38671875( proc.memory.availableMB‚ 1298.8359375 disk.nvme0n1.in‚ 603.58984375 disk.nvme1n1.out‚ 47259.78125 disk.sdb1.in‚0 disk.rbd2.in‚0" proc.memory.rssMB‚ 3001.5703125* proc.memory.percent‚18.320131301879883 network.sent‚ 1.7333854e+07! disk.sdb2.out‚910.51220703125 disk.sda1.in‚0 disk.nvme1n1.in‚ 498.80078125 disk.sdb1.out‚0 disk.md0.out‚ 877.44140625 proc.cpu.threads‚17) disk./.usagePercent‚75.88088885597108 disk.rbd0.in‚0˜šª72¦- _wandbruntime‚7955  _runtime‚7955/ 5ɧà _step‚559 ' lr-AdamW/pg1‚0.00040416209575737594 ' lr-AdamW/pg2‚0.00040416209575737594 trainer/global_step‚6849 % _timestamp‚1.7885183302969022e+09 _runtime‚7955.767256173«v nÎ¨È ' lr-AdamW/pg1‚0.00040416209575737594 ' lr-AdamW/pg2‚0.00040416209575737594 trainer/global_step‚6849 % _timestamp‚1.7885183302969022e+09 _runtime‚7955.767256173 _step‚559¯}6'Œ2©- _wandbruntime‚7955  _runtime‚7955DÙ´³ü'ªöO ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚637444541560320 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 _step‚560 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚87755255 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26088 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917410816 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407872 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 128753356 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚629323568927744 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚629323544987648 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7779200920576 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 312820766 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚631 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚562367898244608 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚87755253 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411420 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚570147099165184 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8120750064128 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407872 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚633 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1212 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 246508544 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 171670782 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚40997470 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 128752723 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚40998101 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 141123896 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7750198463496 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚562367880241664 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7749301052076 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177047040 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 222568448 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007241 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚562367898244608 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 141123896 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955591168 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26088 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 171670782 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚549862800196240 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 trainer/global_step‚6849 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚570146181754368 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7778301512704 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚562367880241664 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214665 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 312794676 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 171696870 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚637444295051776 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚557612998659736 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 141123894 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410816 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955591168 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450780 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1187 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 % _timestamp‚1.7885183302979674e+09 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7778301512704 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚570146181754368 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚549862783156880 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 246508544 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 171670782 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚40997470 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 128752723 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚40998101 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 141123896 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7750198463496 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚562367880241664 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7749301052076 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177047040 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 222568448 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007241 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚562367898244608 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 141123896 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955591168 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26088 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 171670782 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚549862800196240 WJDeviceStatsMonitor.on_train_batch_start/re ²¨³quested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 trainer/global_step‚6849 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚570146181754368 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7778301512704 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚562367880241664 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214665 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 312794676 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 171696870 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚637444295051776 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚557612998659736 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 141123894 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410816 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955591168 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450780 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1187 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 % _timestamp‚1.7885183302979674e+09 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7778301512704 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚570146181754368 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚549862783156880 JDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 171695373 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31686 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 141143792 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 399863808 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955591168 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7779414316032 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚570226924853248 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚87767629 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚637534761585664 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚557692087415390 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007241 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚6849 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26114 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚562447528540160 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7780313737216 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 128770836 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 254883840 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚562447528540160 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411628 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 312839165 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 171695373 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 312865281 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26114 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 141143794 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 230943744 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚41003207 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8121897377280 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177047040 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 312865281 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚570227842277376 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚629412864208384 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7751307125022 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177047040 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7780313737216 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899421184 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚631 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7750409713394 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712L&® '®…N ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚562447528540160 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411628 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 312839165 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 171695373 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 312865281 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26114 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 141143794 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 230943744 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚41003207 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8121897377280 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177047040 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 312865281 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚570227842277376 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚629412864208384 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7751307125022 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177047040 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7780313737216 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899421184 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚631 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7750409713394 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚570227842277376 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚549940763251008 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917424128 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚557691172964402 _step‚561 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 171695373 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31686 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 141143792 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 399863808 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955591168 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7779414316032 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚570226924853248 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚87767629 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚637534761585664 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚557692087415390 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007241 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚6849 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26114 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚562447528540160 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7780313737216 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 128770836 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 254883840 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0±åĵ/2¯- _wandbruntime‚7956  _runtime‚7956ô‰ñøZ°Ô /loss/FRt1/detection_loss‚2.1068839853808945 /loss/FRt4/assignment_loss‚3.658701116388494 2loss/SRqqt4/assignment_loss‚1.5935199427604676 0loss/FBt3/assignment_loss‚0.3174929193221033 _step‚562 0loss/FRt1/assignment_loss‚3.0923657615979514 0loss/FRt2/assignment_loss‚1.3198135616827984 /loss/FRt3/assignment_loss‚2.561881513065762 0loss/FBt1/assignment_loss‚0.3491305849274795 .loss/FBt4/assignment_loss‚0.53883182851132 /loss/FRt3/detection_loss‚2.0890488644440968 /loss/FRt4/detection_loss‚3.3981572795997965 1loss/SRqqt3/detection_loss‚1.7502731970967131 2loss/SRqqt1/assignment_loss‚1.1775267346702674 0loss/FBt2/assignment_loss‚0.1576031962350565 /loss/FRt2/detection_loss‚1.5807277612692239 /loss/FBt1/detection_loss‚2.4067935682833195 -loss/FBt3/detection_loss‚2.36013348698616 /loss/FBt4/detection_loss‚2.6534924134612083 epoch‚5 _runtime‚7956.809314295 1loss/SRqqt1/detection_loss‚1.8192757857923811 /loss/FBt2/detection_loss‚0.5961720467583366 2loss/SRqqt3/assignment_loss‚1.1132180455556893 1loss/SRqqt4/detection_loss‚2.3483586323261263 &loss/total_loss‚41.193141872877774 trainer/global_step‚6849 % _timestamp‚1.7885183313388877e+09 2loss/SRqqt2/assignment_loss‚1.1125233111482473 1loss/SRqqt2/detection_loss‚1.0912163356138813|lµ_±Ù 2loss/SRqqt4/assignment_loss‚1.5935199427604676 0loss/FBt3/assignment_loss‚0.3174929193221033 _step‚562 0loss/FRt1/assignment_loss‚3.0923657615979514 0loss/FRt2/assignment_loss‚1.3198135616827984 /loss/FRt3/assignment_loss‚2.561881513065762 0loss/FBt1/assignment_loss‚0.3491305849274795 .loss/FBt4/assignment_loss‚0.53883182851132 /loss/FRt3/detection_loss‚2.0890488644440968 /loss/FRt4/detection_loss‚3.3981572795997965 1loss/SRqqt3/detection_loss‚1.7502731970967131 2loss/SRqqt1/assignment_loss‚1.1775267346702674 0loss/FBt2/assignment_loss‚0.1576031962350565 /loss/FRt2/detection_loss‚1.5807277612692239 /loss/FBt1/detection_loss‚2.4067935682833195 -loss/FBt3/detection_loss‚2.36013348698616 /loss/FBt4/detection_loss‚2.6534924134612083 epoch‚5 _runtime‚7956.809314295 1loss/SRqqt1/detection_loss‚1.8192757857923811 /loss/FBt2/detection_loss‚0.5961720467583366 2loss/SRqqt3/assignment_loss‚1.1132180455556893 1loss/SRqqt4/detection_loss‚2.3483586323261263 &loss/total_loss‚41.193141872877774 trainer/global_step‚6849 % _timestamp‚1.7885183313388877e+09 2loss/SRqqt2/assignment_loss‚1.1125233111482473 1loss/SRqqt2/detection_loss‚1.0912163356138813 /loss/FRt1/detection_loss‚2.1068839853808945 /loss/FRt4/assignment_loss‚3.658701116388494²|N•Ñ”²‚(:‰ ÿêÔµ¯óÿ gpu.0.gpu‚55.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚36.0 gpu.0.powerWatts‚114.131' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.53275 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ž0Gþ`³‚(:Õ ÿêÔ³­Œ€ disk.nvme1n1.in‚ 498.80078125 disk.nvme1n1.out‚ 47260.125 disk.sdb1.in‚0 disk.rbd2.out‚2.125 cpu‚25.201032687610542 network.recv‚ 2.590087e+06) disk./.usagePercent‚75.88088885597108 disk.rbd2.in‚0 disk.sdb.out‚911.82373046875 disk.sda.out‚911.82373046875 network.sent‚ 1.7339694e+07! disk.sda2.out‚911.82373046875 disk.sdb1.out‚0 disk.md0.in‚5.3125! disk.sdb2.out‚911.82373046875 disk.sda1.out‚0 disk.md0p1.in‚5.3125 disk.sda.in‚ 4.38671875 disk.sda2.in‚ 4.38671875 disk.rbd0.in‚0 disk.sda1.in‚0" proc.memory.rssMB‚ 3001.5703125 proc.cpu.threads‚17 disk.nvme0n1.in‚ 603.58984375* proc.memory.percent‚18.320131301879883 disk.md0.out‚ 878.68359375 disk.sdb2.in‚ 0.92578125# disk.nvme0n1.out‚43004.99609375' proc.memory.availableMB‚ 1298.578125 disk.dm-0.in‚ 1102.390625 disk.sdb.in‚ 0.92578125 disk.rbd0.out‚2.1875 disk.dm-0.out‚90265.12109375 disk.md0p1.out‚ 878.68359375# memory_percent‚92.0741081237793$ disk./.usageGB‚2713.8681335449226uúò”´‚(:‰ Ò¿êÔÙ¿µÿ gpu.0.gpu‚29.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚168.715' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.17875 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ä?HJµ‚(:Æ Ò¿êԲϛ€ network.sent‚ 1.7450064e+07 disk.sda.in‚ 4.38671875 disk.nvme1n1.in‚ 498.80078125 disk.rbd2.out‚2.125 disk.sda1.in‚0 disk.sda1.out‚0 disk.md0p1.in‚5.3125 disk.nvme0n1.in‚ 603.58984375 disk.nvme0n1.out‚43148.75 disk.sda2.in‚ 4.38671875# disk.nvme1n1.out‚47417.52734375 disk.md0.out‚ 879.98046875 network.recv‚ 2.596098e+06 disk.sda.out‚ 913.17578125 disk.sdb1.out‚0 disk.rbd2.in‚0" proc.memory.rssMB‚ 3001.5703125$ disk./.usageGB‚2713.868133544922 disk.sdb.in‚ 0.92578125 diskŠw¶a.md0.in‚5.3125 disk.dm-0.in‚ 1102.390625 disk.dm-0.out‚90566.27734375 cpu‚25.2007169117827) disk./.usagePercent‚75.88088885597108 disk.sdb.out‚ 913.17578125 disk.rbd0.in‚0 disk.sdb2.in‚ 0.92578125# memory_percent‚92.0741081237793 disk.sda2.out‚ 913.17578125 disk.sdb2.out‚ 913.17578125 disk.md0p1.out‚ 879.98046875* proc.memory.percent‚18.320131301879883 disk.rbd0.out‚2.1875 disk.sdb1.in‚0' proc.memory.availableMB‚ 1298.578125 proc.cpu.threads‚17¸±ÙÏ’¶‚(:‡ á¿êÔ¬ øÿ gpu.0.gpu‚2.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚37.0 gpu.0.powerWatts‚160.922' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚40.2305 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0.;‘f·‚(:Û á¿êÔòÔŠ) disk./.usagePercent‚75.88089269575023 disk.sda2.in‚ 4.38671875 disk.rbd0.out‚2.1875 proc.cpu.threads‚17 disk.sdb.in‚ 0.92578125 disk.sda.out‚915.59130859375 disk.rbd0.in‚0 disk.rbd2.in‚0$ memory_percent‚92.07274913787842) proc.memory.availableMB‚ 1298.80078125 disk.dm-0.out‚ 90832.046875 disk.sda.in‚ 4.38671875 disk.md0.in‚5.3125 disk.nvme0n1.in‚ 603.58984375# disk.nvme0n1.out‚43256.76171875 cpu‚25.200429452981346% disk./.usageGB‚2713.8682708740234 disk.nvme1n1.in‚ 498.80078125 network.sent‚ 1.7453118e+07 disk.md0p1.out‚ 882.32421875" proc.memory.rssMB‚ 3001.5703125 disk.sdb1.out‚0 disk.sdb2.in‚ 0.92578125! disk.sdb2.out‚915.59130859375* proc.memory.percent‚18.320131301879883 disk.dm-0.in‚ 1102.390625 disk.md0p1.in‚5.3125 network.recv‚ 2.59943e+06! disk.sda2.out‚915.59130859375# disk.nvme1n1.out‚47575.28515625 disk.sdb1.in‚0 disk.rbd2.out‚2.125 disk.sda1.out‚0 disk.sdb.out‚915.59130859375 disk.md0.out‚ 882.32421875 disk.sda1.in‚0^<{Á“¸‚(:ˆ ð¿êÔ™¹€€ gpu.0.gpu‚55.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚36.0 gpu.0.powerWatts‚122.486' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚30.6215 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ü\1ùa¹‚(:Ö ð¿êÔ½€ž disk.rbd2.out‚2.125 disk.sdb.in‚3.1875 disk.sdb.out‚918.15869140625 disk.sda.out‚918.15869140625 network.sent‚ 1.7461262e+07 disk.rbd2.in‚0 disk.nvme0n1.in‚ 603.58984375$ memory_percent‚92.07158088684082 disk.sdb2.in‚3.1875 disk.sda1.out‚0 disk.dm-0.out‚ 90832.234375 disk.rbd0.out‚2.1875 disk.nvme1n1.out‚ 47575.46875" proc.memory.rssMB‚ 3001.5703125* proc.memory.percent‚18.320131301879883 proc.cpu.threads‚17% disk./.usageGB‚2713.8684272766113 disk.sdb1.in‚0( proc.memory.availableMB‚ 1298.9921875 network.recv‚ 2.603655e+06 disk.md0p1.out‚ 884.83984375! disk.nvme0n1.out‚ 43256.765625 disk.sda2.in‚ 6.08203125! disk.sda2.out‚918.15869140625 disk.rbd0.in‚0 disk.dm-0.in‚ 1102.390625) disk./.usagePercent‚75.88089706883206 disk.md0.out‚ 884.83984375 disk.sda1.in‚0 disk.md0p1.in‚ 9.26953125 disk.sdb1.out‚0 disk.md0.in‚ 9.26953125! disk.sdb2.out‚918.15869140625 disk.sda.in‚ 6.08203125 disk.nvme1n1.in‚ 498.80078125 cpu‚25.200147609550925Šå-µ2º- _wandbruntime‚8014  _runtime‚8014½K$ɻà trainer/global_step‚6899 % _timestamp‚1.7885183894981143e+09 _runtime‚8014.969960171 _step‚563 ' lr-AdamW/pg1‚0.00039798420747673576 ' lr-AdamW/pg2‚0.00039798420747673576¬b¾Î¼È trainer/global_step‚6899 % _timestamp‚1.7885183894981143e+09 _runtime‚8014.969960171 _step‚563 ' lr-AdamW/pg1‚0.00039798420747673576 ' lr-AdamW/pg2‚0.00039798420747673576³q$G¯2½- _wandbruntime‚8014  _runtime‚80146 Ù8û'¾õO XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚88374055 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚41283182 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 142118796 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 142118796 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚574184268523008 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407872 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 trainer/global_step‚6899 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917410816 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410816 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1212 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚566349413022208 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚574183351112192 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 399863808 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 172929772 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚41283789 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 315022478 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚609 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 248605696 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26088 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚633789529964544 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214665 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177047040 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7834855500800 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 129657235 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411420 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 $ _timestamp‚1.788518389503023e+09 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7833956092928 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407872 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚561567450023244 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 172929772 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 142118794 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955591168 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚641968642983936 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚553761804902640 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7804747709184 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7834855500800 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚566349395019264 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚574183351112192 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 315048568 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 142118794 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚607 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚641968394378240 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450780 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 _step‚564 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 172929772 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 142118794 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955591168 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚641968642983936 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚553761804902640 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7804747709184 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7834855500800 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚566349395019264 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚574183351112192 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 315048568 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 142118794 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚607 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚641968394378240 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450780 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 _step‚564 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 trainer/global_step‚6899 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917410816 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410816 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1212 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚566349413022208 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚574183351112192 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 399863808 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 172929772 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚41283789 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 315022478 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚609 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 248605696 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26088 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚633789529964544 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214665 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177047040 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7834855500800 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 129657235 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411420 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 $ _timestamp‚1.788518389503023e+09 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7833956092928 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407872 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚561567450023244 JDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 129675307 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 172954389 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 214166528 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚574265011635200 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚642059139741184 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 172928275 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 315093083 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚553839784996768 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 142138694 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚566429043317760 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚553839767957408 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚566429043317760 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7835968317440 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177047040 TEDeviªNñ–|ceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007241 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 315066967 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 142138692 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31686 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚88386429 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1187 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚561646538778898 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚566429025314816 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 399863808 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8180290555904 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 315093083 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 142138692 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450988ÝÐ]Ë 'Â…N E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚566429043317760 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7835968317440 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177047040 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007241 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 315066967 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 142138692 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31686 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚88386429 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1187 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚561646538778898 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚566429025314816 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 399863808 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8180290555904 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 315093083 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 142138692 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450988 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 129675307 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 172954389 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 214166528 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚574265011635200 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚642059139741184 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 172928275 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 315093083 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚553839784996768 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 142138694 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚566429043317760 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚553839767957408µÖ©Â2Ã- _wandbruntime‚8016  _runtime‚8016‰-²5ZÄÔ 1loss/FBt1/assignment_loss‚0.22578985219127692 1loss/FBt3/assignment_loss‚0.08628240293182898 0loss/FBt4/assignment_loss‚0.2582346768227911 .loss/FBt3/detection_loss‚3.530941390991211 2loss/SRqqt2/assignment_loss‚1.0574913433261424 /loss/FRt2/detection_loss‚1.5057550988780954 .loss/FBt1/detection_loss‚2.372038555145264 /loss/FBt2/detection_loss‚0.7585463081118896 _runtime‚8016.181713115 1loss/SRqqt3/assignment_loss‚1.291889746016001 1loss/SRqqt3/detection_loss‚1.3559182013439781 /loss/FBt4/detection_loss‚2.5316703861409966 trainer/global_step‚6899 $ _timestamp‚1.788518390711588e+09 2loss/SRqqt4/assignment_loss‚1.3839536676964452 .loss/FRt3/detection_loss‚2.360881940027078 .loss/FRt4/detection_loss‚2.857369207910129 &loss/total_loss‚39.497762995106356 0loss/FRt2/assignment_loss‚1.1463200251261394 0loss/FRt3/assignment_loss‚2.3496564507484434 1loss/SRqqt1/detection_loss‚1.6070707008546712 _step‚566 0loss/FRt1/assignment_loss‚3.1661546244507743 2loss/SRqqt1/assignment_loss‚1.3685718598794572 /loss/FRt1/detection_loss‚2.0781225385004736 /loss/FRt4/assignment_loss‚2.682043135166168 0loss/FBt2/assignment_loss‚0.1278411242752086 1loss/SRqqt2/detection_loss‚0.9456032119794853 0loss/SRqqt4/detection_loss‚2.449616546592405 epoch‚5“ì2 _ÅÙ 1loss/SRqqt3/assignment_loss‚1.291889746016001 1loss/SRqqt3/detection_loss‚1.3559182013439781 /loss/FBt4/detection_loss‚2.5316703861409966 trainer/global_step‚6899 $ _timestamp‚1.788518390711588e+09 2loss/SRqqt4/assignment_loss‚1.3839536676964452 .loss/FRt3/detection_loss‚2.360881940027078 .loss/FRt4/detection_loss‚2.857369207910129 &loss/total_loss‚39.497762995106356 0loss/FRt2/assignment_loss‚1.1463200251261394 0loss/FRt3/assignment_loss‚2.3496564507484434 1loss/SRqqt1/detection_loss‚1.6070707008546712 _step‚566 0loss/FRt1/assignment_loss‚3.1661546244507743 2loss/SRqqt1/assignment_loss‚1.3685718598794572 /loss/FRt1/detection_loss‚2.0781225385004736 /loss/FRt4/assignment_loss‚2.682043135166168 0loss/FBt2/assignment_loss‚0.1278411242752086 1loss/SRqqt2/detection_loss‚0.9456032119794853 0loss/SRqqt4/detection_loss‚2.449616546592405 epoch‚5 1loss/FBt1/assignment_loss‚0.22578985219127692 1loss/FBt3/assignment_loss‚0.08628240293182898 0loss/FBt4/assignment_loss‚0.2582346768227911 .loss/FBt3/detection_loss‚3.530941390991211 2loss/SRqqt2/assignment_loss‚1.0574913433261424 /loss/FRt2/detection_loss‚1.5057550988780954 .loss/FBt1/detection_loss‚2.372038555145264 /loss/FBt2/detection_loss‚0.7585463081118896 _runtime‚8016.181713115¶0B o”Æ‚(:‰ ÿ¿êÔŽªþ gpu.0.gpu‚40.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚164.989' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚41.24725 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÝÉPí[Ç‚(:Ð ÿ¿êÔÁ”ßþ$ disk./.usageGB‚2712.814971923828 disk.dm-0.out‚90833.28515625 disk.md0p1.out‚ 886.40234375 disk.sda.out‚ 919.76953125 disk.sda2.out‚ 919.76953125 disk.rbd2.in‚0) proc.memory.availableMB‚ 1298.72265625 network.sent‚ 1.7464929e+07 disk.md0.out‚ 886.40234375 disk.sdb2.out‚ 919.76953125 proc.cpu.threads‚17 disk.rbd0.out‚2.1875 disk.sdb1.out‚0 disk.sda2.in‚ 6.08203125 network.recv‚ 2.607626e+06 disk.sdb.in‚3.1875 disk.rbd2.out‚2.125 disk.sdb2.in‚3.1875* proc.memory.percent‚18.320131301879883 cpu‚25.199790888773244 disk.sda1.out‚0 disk.dm-0.in‚ 1102.390625 disk.sda.in‚ 6.08203125 disk.nvme1n1.in‚ 498.80078125# disk.nvme1n1.out‚47576.40234375$ memory_percent‚92.07322597503662 disk.sda1.in‚0 disk.nvme0n1.in‚ 603.58984375 disk.sdb.out‚ 919.76953125 disk.md0.in‚ 9.26953125" proc.memory.rssMB‚ 3001.5703125 disk.md0p1.in‚ 9.26953125" disk.nvme0n1.out‚ 43256.8828125 disk.rbd0.in‚0 disk.sdb1.in‚0) disk./.usagePercent‚75.85144201626292·Fô§”È‚(:‰ ŽÀêÔöŠÙý gpu.0.gpu‚40.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚170.891' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.72275 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‡{i`É‚(:Õ ŽÀêÔöÿþ network.sent‚ 1.7573202e+07 disk.rbd0.out‚2.1875 disk.rbd2.in‚0 disk.sda1.in‚0 disk.dm-0.in‚ 1102.390625# disk.nvme0n1.out‚43400.63671875 disk.sda.in‚ 6.08203125 cpu‚25.199479676155523 disk.sdb2.out‚921.7021484375 disk.dm-0.out‚91064.66015625 disk.nvme0n1.in‚ 603.58984375 disk.md0.out‚ 888.27734375$ memory_percent‚91.55185222625732 disk.sda1.out‚0 disk.nvme1n1.in‚ 498.80078125 disk.sdb1.in‚0$ disk./.usageGB‚2712.814556121826 disk.sdb2.in‚3.1875 disk.md0p1.in‚ 9.26953125 disk.md0p1.out‚ 888.27734375 disk.rbd0.in‚0 proc.memory.rssMB‚ 2912.71875 disk.rbd2.out‚2.125 disk.sdb.in‚3.1875 disk.sdb.out‚921.7021484375 disk.sda.out‚921.7021484375 disk.sdb1.out‚0 proc.cpu.threads‚17 network.recv‚ 2.613981e+06 disk.sda2.in‚ 6.08203125) proc.memory.availableMB‚ 1384.14453125) proc.memory.percent‚17.77782440185547) disk./.usagePercent‚75.85143039026492 disk.sda2.out‚921.7021484375" disk.nvme1n1.out‚ 47664.0234375 disk.md0.in‚ 9.26953125ùšžÊ‚(:“ ÀêÔÊÝŒ€ gpu.0.gpu‚56.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚37.0 gpu.0.powerWatts‚119.368' gpu.0.enforcedPowerLimitWatts‚400.0) gpu.0.powerPercent‚29.841999999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ð0ÝRË‚(:Ç ÀêÔäñ©€ disk.sdb1.in‚0 disk.sda1.in‚0 network.recv‚ 2.617349e+06 disk.sda2.out‚ 923.1953125 disk.nvme1n1.in‚ 498.80078125# disk.nvme1n1.out‚47664.30859375 disk.sdb2.out‚ 923.1953125) disk./.usagePercent‚75.83662110868248% disk./.usageGB‚2712.2849044799805 disk.md0.in‚ 9.26953125$ memory_percent‚91.55313968658447 cpu‚25.199162589695888 disk.nvme0n1.in‚ 603.58984375! disk.nvme0n1.out‚ 43400.796875 disk.rbd0.in‚0 disk.sdb1.out‚0 disk.md0.out‚ 889.7265625 disk.rbd2.in‚0 disk.dm-0.in‚ 1102.390625 disk.sda.in‚ 6.08203125 disk.sda.out‚ 923.1953125) proc.memory.percent‚17.77782440185547 disk.sda2.in‚ 6.08203125 disk.md0p1.in‚ 9.26953125 disk.sdb.in‚3.1875 disk.rbd0.out‚2.1875 disk.sdb2.in‚3.1875 disk.sda1.out‚0) proc.memory.availableMB‚ 1383.93359375 proc.cpu.threads‚17 disk.rbd2.out‚2.125 network.sent‚ 1.7578441e+07 disk.dm-0.out‚91065.10546875 proc.memory.rssMB‚ 2912.71875 disk.md0p1.out‚ 889.7265625 disk.sdb.out‚ 923.1953125|ÄM2Ì- _wandbruntime‚8067  _runtime‚8067ñW‹ÇÍÁ _runtime‚8067.707651325 _step‚567 & lr-AdamW/pg1‚0.0003918127516435564 & lr-AdamW/pg2‚0.0003918127516435564 trainer/global_step‚6949 % _timestamp‚1.7885184422373261e+09ÚÊ{½ÌÎÆ & lr-AdamW/pg1‚0.0003918127516435564 & lr-AdamW/pg2‚0.0003918127516435564 trainer/global_step‚6949 % _timestamp‚1.7885184422373261e+09 _runtime‚8067.707651325 _step‚567·i]2Ï- _wandbruntime‚8067  _runtime‚80672­Ë<ú'ÐôO ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚570330927799808 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955591168 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7860194366292 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 399863808 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚578220520470016 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 143113696 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚565521901386752 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 174162674 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26088 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 174136586 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚622 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450780 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚557660792569680 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 317250280 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955591168 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007241 trainer/global_step‚6949 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 130561865 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚578221437880832 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1212 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚570330927799808 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 174136586 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚88992853 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8237344475648 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 143113694 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚88992855 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚620 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚41569012 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚646492835476992 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407872 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 $ _timestamp‚1.788518442238378e+09 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚570330909796864 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917410816 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚578220520470016 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 143113694 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 143113696 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1187 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400Yç)oS ÑùO UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407872 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 $ _timestamp‚1.788518442238378e+09 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚570330909796864 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917410816 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚578220520470016 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 143113694 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 143113696 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1187 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚570330927799808 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955591168 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7860194366292 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 399863808 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚578220520470016 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 143113696 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚565521901386752 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 174162674 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26088 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 174136586 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚622 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450780 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚557660792569680 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 317250280 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955591168 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007241 trainer/global_step‚6949 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 130561865 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚578221437880832 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1212 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚570330927799808 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 174136586 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚88992853 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8237344475648 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 143113694 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚88992855 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚620 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚41569012 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚646492835476992 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728¸G,Ž 2Ò- _wandbruntime‚8068  _runtime‚8068`eÅ'ÓÿM OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 143133592 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 399863808 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 143133594 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 317320885 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8238512705024 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚578302180993024 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 240203776 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚557738789703168 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7861303027610 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007241 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 317294769 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 174187291 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚570410540092416 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899421184 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚608 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1187 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 trainer/global_step‚6949 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚578302180993024 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚638344810222080 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 317320885 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚ 8068.66454598 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚565600990142406 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8238296441344 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚557738772663808 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚578301263568896 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚41575295 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177047040 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚646583322927104 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 174161177 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 174161177 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚638344786281984 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450988 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚565600075691418 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 143133592 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚570410558095360 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚41574687 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917424128 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 317294769 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7891622897664 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 130580526 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7890723476480 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26114 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7890723476480 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚89005229 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7862200439238 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 174187291 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7891622897664 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 130579916 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177047040 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31686 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411628 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚89005231 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 317320885 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚ 8068.66454598 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚565600990142406 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8238296441344 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚557738772663808 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚578301263568896 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚41575295 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177047040 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚646583322927104 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 174161177 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 174161177 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚638344786281984 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450988 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚565600075691418 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 143133592 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚570410558095360 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚41574687 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917424128 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 317294769 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7891622897664 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 130580526 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7890723476480 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26114 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7890723476480 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚89005229 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7862200439238 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 174187291 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7891622897664 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 130579916 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177047040 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31686 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411628 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚89005231 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 317320885 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8238512705024 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚578302180993024 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 240203776 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚557738789703168 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7861303027610 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007241 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 317294769 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 174187291 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚570410540092416 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899421184 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚608 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1187 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 trainer/global_step‚6949 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128¹¼íä2Õ- _wandbruntime‚8068  _runtime‚8068Q/’î[ÖÕ 1loss/SRqqt4/detection_loss‚2.3296837672110526 /loss/FBt3/detection_loss‚1.9916184365749359 .loss/FRt1/detection_loss‚1.812517159287871 2loss/SRqqt3/assignment_loss‚1.6209389787433404 % _timestamp‚1.7885184431961062e+09 2loss/SRqqt4/assignment_loss‚1.4916986486604136 1loss/SRqqt3/detection_loss‚1.3657656716981104 .loss/FBt2/detection_loss‚0.540857459406058 0loss/FRt4/assignment_loss‚3.2599729983245624 1loss/FBt1/assignment_loss‚0.13337127481402908 1loss/FBt4/assignment_loss‚0.28380910464976394 .loss/FRt3/detection_loss‚2.033184407173463 /loss/FBt1/detection_loss‚2.3399548133214316 epoch‚5 0loss/FRt2/assignment_loss‚1.2812476706504823 /loss/FRt3/assignment_loss‚2.465734989013312 .loss/FBt4/detection_loss‚2.265045086542765 trainer/global_step‚6949 /loss/FRt2/detection_loss‚1.6550739791803062 1loss/SRqqt1/assignment_loss‚1.427282480028819 1loss/FBt2/assignment_loss‚0.20324564869868672 1loss/SRqqt1/detection_loss‚1.6911887417034226 _runtime‚8068.666225467 _step‚570 /loss/FRt1/assignment_loss‚2.594593937580402 .loss/FRt4/detection_loss‚2.581175954902873 1loss/SRqqt2/detection_loss‚1.1665985998460802 %loss/total_loss‚37.75340917602454 2loss/SRqqt2/assignment_loss‚1.0032944239328725 1loss/FBt3/assignment_loss‚0.21555494407948572†®“`×Ú 0loss/FRt4/assignment_loss‚3.2599729983245624 1loss/FBt1/assignment_loss‚0.13337127481402908 1loss/FBt4/assignment_loss‚0.28380910464976394 .loss/FRt3/detection_loss‚2.033184407173463 /loss/FBt1/detection_loss‚2.3399548133214316 epoch‚5 0loss/FRt2/assignment_loss‚1.2812476706504823 /loss/FRt3/assignment_loss‚2.465734989013312 .loss/FBt4/detection_loss‚2.265045086542765 trainer/global_step‚6949 /loss/FRt2/detection_loss‚1.6550739791803062 1loss/SRqqt1/assignment_loss‚1.427282480028819 1loss/FBt2/assignment_loss‚0.20324564869868672 1loss/SRqqt1/detection_loss‚1.6911887417034226 _runtime‚8068.666225467 _step‚570 /loss/FRt1/assignment_loss‚2.594593937580402 .loss/FRt4/detection_loss‚2.581175954902873 1loss/SRqqt2/detection_loss‚1.1665985998460802 %loss/total_loss‚37.75340917602454 2loss/SRqqt2/assignment_loss‚1.0032944239328725 1loss/FBt3/assignment_loss‚0.21555494407948572 1loss/SRqqt4/detection_loss‚2.3296837672110526 /loss/FBt3/detection_loss‚1.9916184365749359 .loss/FRt1/detection_loss‚1.812517159287871 2loss/SRqqt3/assignment_loss‚1.6209389787433404 % _timestamp‚1.7885184431961062e+09 2loss/SRqqt4/assignment_loss‚1.4916986486604136 1loss/SRqqt3/detection_loss‚1.3657656716981104 .loss/FBt2/detection_loss‚0.540857459406058ºž ’Ø‚(:‡ ¬ÀêÔ’Ôýÿ gpu.0.gpu‚38.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚180.196' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚45.049 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0)æ*G]Ù‚(:Ò ¬ÀêÔšÛЀ disk.dm-0.out‚91126.83984375 disk.md0p1.out‚ 891.24609375 disk.sda.out‚924.7705078125 disk.dm-0.in‚ 1102.390625 disk.sda2.in‚ 6.08203125 disk.nvme1n1.in‚ 498.80078125 disk.sdb1.in‚0 disk.md0.out‚ 891.24609375 disk.rbd0.out‚2.1875" disk.nvme1n1.out‚ 47665.4921875$ memory_percent‚91.55340194702148' proc.memory.availableMB‚ 1383.890625 disk.sdb2.in‚3.1875 disk.sda1.in‚0 disk.nvme0n1.in‚ 603.58984375# disk.nvme0n1.out‚43461.34765625 disk.sdb1.out‚0 proc.memory.rssMB‚ 2912.71875 cpu‚25.19885328368454 disk.rbd2.in‚0 disk.sdb2.out‚924.7705078125 disk.md0p1.in‚ 9.26953125 disk.sda.in‚ 6.08203125 disk.sda2.out‚924.7705078125 disk.rbd0.in‚0 network.sent‚ 1.7584471e+07 network.recv‚ 2.621796e+06 disk.md0.in‚ 9.26953125 disk.sdb.in‚3.1875 disk.sdb.out‚924.7705078125) proc.memory.percent‚17.77782440185547) disk./.usagePercent‚75.83407480179913$ disk./.usageGB‚2712.193836212158 disk.sda1.out‚0 proc.cpu.threads‚17 disk.rbd2.out‚2.125­*”Ú‚(:‰ »ÀêÔÙ¨Úÿ gpu.0.gpu‚45.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚152.899' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚38.22475 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Õ£i(VÛ‚(:½ »ÀêÔÓì•€" disk.nvme0n1.out‚ 43533.5234375 disk.sdb.out‚ 926.9453125 disk.sdb1.out‚0 disk.sda1.out‚0) proc.memory.percent‚17.77782440185547% disk./.usageGB‚2712.3346672058105 disk.md0p1.out‚ 893.359375 disk.rbd0.out‚2.1875 disk.rbd2.in‚0 disk.dm-0.in‚ 1102.390625) disk./.usagePercÎ,³òent‚75.83801249532401 disk.sda.in‚ 6.08203125 disk.md0.out‚ 893.359375$ memory_percent‚91.55035018920898 disk.sdb.in‚3.1875 disk.rbd2.out‚2.125 disk.nvme0n1.in‚ 603.58984375 proc.cpu.threads‚17 network.recv‚ 2.62605e+06 disk.md0p1.in‚ 9.26953125 disk.sdb1.in‚0 disk.sdb2.in‚3.1875 disk.rbd0.in‚0 disk.md0.in‚ 9.26953125 disk.sdb2.out‚ 926.9453125 disk.sda1.in‚0 disk.dm-0.out‚ 91199.1953125' proc.memory.availableMB‚ 1384.390625 proc.memory.rssMB‚ 2912.71875 disk.sda.out‚ 926.9453125 disk.sda2.in‚ 6.08203125 disk.sda2.out‚ 926.9453125 disk.nvme1n1.in‚ 498.80078125! disk.nvme1n1.out‚ 47665.671875 cpu‚25.198519353037 network.sent‚ 1.7693392e+07w‚ª¦’Ü‚(:‡ ÊÀêÔŠØëý gpu.0.gpu‚56.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚36.0 gpu.0.powerWatts‚114.408' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.602 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÖdWÝ‚(:Ì ÊÀêÔ˜ƒƒÿ disk.dm-0.out‚91200.15234375 disk.sdb2.in‚3.1875 disk.sdb.in‚3.1875 disk.sda2.out‚928.5439453125 disk.sdb1.out‚0 proc.memory.rssMB‚ 2912.71875) proc.memory.percent‚17.77782440185547 disk.md0p1.in‚ 9.26953125 disk.sdb1.in‚0% disk./.usageGB‚2712.3172340393066 disk.rbd2.in‚0 disk.sdb2.out‚928.5439453125 disk.nvme0n1.in‚ 603.58984375 disk.sda.out‚928.5439453125 proc.cpu.threads‚17 disk.nvme0n1.out‚ 43534.3125 disk.nvme1n1.in‚ 498.80078125# disk.nvme1n1.out‚47665.83984375' proc.memory.availableMB‚ 1383.859375 network.recv‚ 2.630885e+06 disk.dm-0.in‚ 1102.390625 disk.sdb.out‚928.5439453125 disk.rbd0.in‚0 disk.rbd2.out‚2.125 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda.in‚ 6.08203125 disk.rbd0.out‚2.1875$ memory_percent‚91.55359268188477 cpu‚25.198221862060095( disk./.usagePercent‚75.8375250566922 disk.md0.in‚ 9.26953125 disk.md0p1.out‚ 894.890625 disk.sda2.in‚ 6.08203125 network.sent‚ 1.7697509e+07 disk.md0.out‚ 894.890625²Z’Þ‚(:‡ ÙÀêԼשÿ gpu.0.gpu‚56.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚39.0 gpu.0.powerWatts‚173.676' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.419 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0zcß‚(:Ø ÙÀêÔû¼Ðÿ disk.rbd0.out‚2.1875 disk.rbd2.out‚2.125) proc.memory.availableMB‚ 1384.10546875 disk.sdb2.in‚3.1875 disk.sda.in‚ 6.08203125 disk.sda.out‚930.16162109375! disk.sda2.out‚930.16162109375 disk.sdb1.out‚0 disk.md0p1.in‚ 9.26953125 disk.md0p1.out‚ 896.44921875 disk.nvme0n1.in‚ 603.58984375 disk.rbd0.in‚0 network.sent‚ 1.7702734e+07 network.recv‚ 2.634032e+06) disk./.usagePercent‚75.83888647172421! disk.sdb2.out‚930.16162109375 disk.sda1.in‚0 disk.dm-0.out‚91364.40234375 disk.nvme1n1.in‚ 498.80078125# disk.nvme1n1.out‚47666.17578125$ disk./.usageGB‚2712.365924835205 disk.sda1.out‚0 disk.sdb.in‚3.1875 disk.sdb.out‚930.16162109375$ memory_percent‚91.55209064483643 cpu‚25.19791410059114 disk.sda2.in‚ 6.08203125 disk.md0.in‚ 9.26953125 disk.dm-0.in‚ 1102.390625" disk.nvme0n1.out‚ 43698.2265625 disk.md0.out‚ 896.44921875 disk.rbd2.in‚0 proc.memory.rssMB‚ 2912.71875) proc.memory.percent‚17.77782440185547 proc.cpu.threads‚17 disk.sdb1.in‚0.‡I2à- _wandbruntime‚8120  _runtime‚8120è9½ÇáÁ trainer/global_step‚6999 % _timestamp‚1.7885184949376302e+09 _runtime‚8120.407988064 _step‚571 & lr-AdamW/pg1‚0.0003856490143799836 & lr-AdamW/pg2‚0.0003856490143799836Úý*ÌâÆ & lr-AdamW/pg1‚0.0003856490143799836 & lr-AdamW/pg2‚0.0003856490143799836 trainer/global_step‚6999 % _timestamp‚1.7885184949376302e+09 _runtime‚8120.407988064 _step‚571»Zð2ã- _wandbruntime‚8120  _runtime‚8120rv±ü'äöO GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26088 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 399863808 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 175369488 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚642721452038144 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 % _timestamp‚1.7885184949383996e+09 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1212 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26090 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 144108594 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 131467238 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1187 _step‚572 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214665 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 175369488 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚617 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚651016932185600 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚651016681482752 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚574312442577408 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚642721428098048 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 144108596 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7946164661248 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917410816 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 144108594 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 131467857 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 226762752 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚569476352750260 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411420 trainer/global_step‚6999 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955591168 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚582258607238656 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚582257689827840 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955591168 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7945265253376 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 175395576 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚582257689827840 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 319504172 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚561559797276080 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7946164661248 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177047040 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8295253384704 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7915641023400 _runtime‚8120.408991349 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚89611653 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26088 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚574312424574464 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214665 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 175369488 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚617 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚651016932185600 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚651016681482752 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚574312442577408 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚642721428098048 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 144108596 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7946164661248 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917410816 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 144108594 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 131467857 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 226762752 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚569476352750260 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411420 trainer/global_step‚6999 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955591168 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚582258607238656 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚582257689827840 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955591168 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7945265253376 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 175395576 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚582257689827840 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 319504172 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚561559797276080 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7946164661248 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177047040 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8295253384704 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7915641023400 _runtime‚8120.408991349 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚89611653 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26088 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚574312424574464 JY¯ç€N WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚569555441505914 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26114 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚41861263 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚642810747318784 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚569554527054926 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 319548687 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚574392072872960 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚582339350350848 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 144128494 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 144128492 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899421184 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177047040 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8296415433216 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007241 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚574392072872960 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 399863808 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 144128494 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899421184 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955591168 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917424128 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 319548687 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7917647096346 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26114 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚89624031 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚6999 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 175420193 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚651107162752000 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚600 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚89624029 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚598 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 175394079 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 % _timestamp‚1.7885184959059174e+09 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8296642182656 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 175394079 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411628 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 319522571 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚41861861 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 _runtime‚8121.376795547 _step‚573 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 175420193 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 131485292 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 226749440 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚651107413441536 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚574392054870016 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7916749684718 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚582339350350848 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 144128492 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 UHDevÕ†û–WiceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7946378056704 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚642810771258880 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177047040 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚574392054870016 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 319548687 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7917647096346 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26114 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚89624031 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚6999 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 175420193 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚651107162752000 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚600 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚89624029 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚598 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 175394079 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 % _timestamp‚1.7885184959059174e+09 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8296642182656 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 175394079 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411628 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 319522571 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚41861861 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 _runtime‚8121.376795547 _step‚573 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 175420193 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 131485292 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 226749440 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚651107413441536 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚574392054870016 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7916749684718 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚582339350350848 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 144128492 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7946378056704 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚642810771258880 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177047040 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚574392054870016 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26114 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚41861263 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚642810747318784 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚569554527054926 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 319548687 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚574392072872960 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚582339350350848½óÎ^u2é- _wandbruntime‚8121  _runtime‚8121˜ú-[^êØ /loss/FBt2/detection_loss‚0.5924475600870399 trainer/global_step‚6999 0loss/FRt1/assignment_loss‚3.6178667931943327 2loss/SRqqt2/assignment_loss‚0.8811298656167791 /loss/FBt3/detection_loss‚3.0553464740514755 0loss/FRt2/assignment_loss‚1.3728854715824128 2loss/SRqqt1/assignment_loss‚1.1803113093430346 2loss/SRqqt3/assignment_loss‚1.1537742045235961 /loss/FRt3/detection_loss‚2.0388643546351073 epoch‚5 0loss/FRt4/assignment_loss‚3.0953236313966603 .loss/FBt4/detection_loss‚2.786620193719864 % _timestamp‚1.7885184959076827e+09 2loss/SRqqt4/assignment_loss‚1.5685303309134075 1loss/SRqqt2/detection_loss‚1.0447092754766345 /loss/FBt1/detection_loss‚2.1881782694866785 _step‚574 /loss/FRt2/detection_loss‚1.6166476406902075 1loss/SRqqt1/detection_loss‚1.9328142915937034 /loss/SRqqt3/detection_loss‚1.45087544431221 /loss/FRt1/detection_loss‚2.3424068017241995 0loss/FBt3/assignment_loss‚0.1378074501849369 1loss/FBt4/assignment_loss‚0.16437633030527649 1loss/FBt2/assignment_loss‚0.13781286201169426 0loss/SRqqt4/detection_loss‚2.122910615376064 %loss/total_loss‚39.87061538406235 _runtime‚8121.377803132 0loss/FRt3/assignment_loss‚2.1805460350266817 1loss/FBt1/assignment_loss‚0.16025856000028157 /loss/FRt4/detection_loss‚3.0481716188100667µHlcëÝ 2loss/SRqqt1/assignment_loss‚1.1803113093430346 2loss/SRqqt3/assignment_loss‚1.1537742045235961 /loss/FRt3/detection_loss‚2.0388643546351073 epoch‚5 0loss/FRt4/assignment_loss‚3.0953236313966603 .loss/FBt4/detection_loss‚2.786620193719864 % _timestamp‚1.7885184959076827e+09 2loss/SRqqt4/assignment_loss‚1.5685303309134075 1loss/SRqqt2/detection_loss‚1.0447092754766345 /loss/FBt1/detection_loss‚2.1881782694866785 _step‚574 /loss/FRt2/detection_loss‚1.6166476406902075 1loss/SRqqt1/detection_loss‚1.9328142915937034 /loss/SRqqt3/detection_loss‚1.45087544431221 /loss/FRt1/detection_loss‚2.3424068017241995 0loss/FBt3/assignment_loss‚0.1378074501849369 1loss/FBt4/assignment_loss‚0.16437633030527649 1loss/FBt2/assignment_loss‚0.13781286201169426 0loss/SRqqt4/detection_loss‚2.122910615376064 %loss/total_loss‚39.87061538406235 _runtime‚8121.377803132 0loss/FRt3/assignment_loss‚2.1805460350266817 1loss/FBt1/assignment_loss‚0.16025856000028157 /loss/FRt4/detection_loss‚3.0481716188100667 /loss/FBt2/detection_loss‚0.5924475600870399 trainer/global_step‚6999 0loss/FRt1/assignment_loss‚3.6178667931943327 2loss/SRqqt2/assignment_loss‚0.8811298656167791 /loss/FBt3/detection_loss‚3.0553464740514755 0loss/FRt2/assignment_loss‚1.3728854715824128¾À5H’ì‚(:‡ èÀêÔÚë¢ÿ gpu.0.gpu‚31.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚116.916' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚29.229 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0<¥'AVí‚(:Ë èÀêÔý¹Çÿ disk.rbd0.out‚2.1875 disk.rbd2.in‚0 disk.sdb2.in‚3.1875 disk.dm-0.out‚ 91369.0234375 disk.md0p1.in‚ 9.26953125 disk.sda2.in‚ 6.08203125 disk.sda1.in‚0" disk.nvme0n1.out‚ 43702.6484375 disk.sdb.out‚931.6123046875 disk.rbd2.out‚2.125 disk.nvme0n1.in‚ 603.58984375 proc.memory.rssMB‚ 2912.71875) proc.memory.percent‚17.77782440185547 disk.sda.in‚ 6.08203125 disk.md0.in‚ 9.26953125 disk.nvme1n1.in‚ 498.80078125 disk.nvme1n1.out‚ 47666.375 disk.sdb1.in‚0 disk.dm-0.in‚ 1102.390625 disk.md0p1.out‚ 897.85546875 disk.sdb.in‚3.1875# memory_percent‚91.5522813796997) proc.memory.availableMB‚ 1384.07421875 network.recv‚ 2.638661e+06( disk./.usagePercent‚75.8389667871049 disk.sda.out‚931.6123046875 disk.sdb1.out‚0 proc.cpu.threads‚17 network.sent‚ 1.7708859e+07 disk.sda1.out‚0$ disk./.usageGB‚2712.368797302246 disk.sda2.out‚931.6123046875 disk.rbd0.in‚0 disk.md0.out‚ 897.85546875 disk.sdb2.out‚931.6123046875 cpu‚25.1976118309064ûó\(‘î‚(:† ÷ÀêÔ½ùþ gpu.0.gpu‚45.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚37.0 gpu.0.powerWatts‚114.74' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.685 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0…‘\_ï‚(:Ô ÷ÀêÔŒèÛÿ disk.sdb.out‚933.7822265625 disk.sda2.in‚ 6.08203125 disk.rbd0.out‚2.1875 disk.sdb1.out‚0 disk.sda2.out‚933.7822265625! disk.nvme1n1.out‚ 47666.921875 disk.md0.in‚ 9.26953125 disk.sda1.out‚0) proc.memory.availableMB‚ 1383.84765625 cpu‚25.197278858433744 disk.sda.in‚ 6.08203125 disk.sdb2.in‚3.1875 disk.md0p1.out‚ 899.95703125 disk.rbd0.in‚0 disk.nvme1n1.in‚ 498.80078125# memory_percent‚91.5536642074585 proc.memory.rssMB‚ 2912.71875 disk.sdb1.in‚0 disk.sda1.in‚0 proc.cpu.threads‚17% disk./.usageGB‚2712.4957389831543# disk.nvme0n1.out‚43706.13671875 disk.sdb.in‚3.1875 disk.md0.out‚ 899.95703125 disk.sdb2.out‚933.7822265625 disk.dm-0.in‚ 1102.390625 disk.md0p1.in‚ 9.26953125 network.sent‚ 1.7816554e+07 network.recv‚ 2.643961e+06) disk./.usagePercent‚75.84251612963232 disk.nvme0n1.in‚ 603.58984375 disk.rbd2.in‚0 disk.rbd2.out‚2.125 disk.dm-0.out‚91373.05859375 disk.sda.out‚933.7822265625) proc.memory.percent‚17.77782440185547 Zz‘Yð‚(:Î †ÁêÔëæ¸ÿ disk.dm-0.in‚ 1102.390625 disk.dm-0.out‚91497.50390625% proc.memory.availableMB‚ 1383.6875 disk.md0.in‚ 9.26953125 disk.sdb.out‚935.27880859375 proc.memory.rssMB‚ 2912.71875 disk.nvme1n1.in‚ 498.80078125 disk.sdb1.in‚0 disk.sdb1.out‚0! disk.sdb2.out‚935.27880859375 disk.sda1.in‚0% disk./.usageGB‚2712.7459678649902! disk.sda2.out‚935.27880859375 disk.md0.out‚ 901.390625 disk.sdb2.in‚3.1875 disk.nvme0n1.in‚ 603.58984375 disk.nvme0n1.out‚ 43830.4375) proc.memory.percent‚17.77782440185547 disk.sda.in‚ 6.08203125 proc.cpu.threads‚17) disk./.usagePercent‚75.84951263389736 disk.rbd2.in‚0$ memory_percent‚91.55464172363281 disk.sda1.out‚0 disk.md0p1.out‚ 901.390625 network.recv‚ 2.648776e+06 disk.rbd0.in‚0# disk.nvme1n1.out‚47667.06640625 cpu‚25.19701053804309 network.sent‚ 1.7825036e+07 disk.sda2.in‚ 6.08203125 disk.rbd0.out‚2.1875 disk.rbd2.out‚2.125 disk.md0p1.in‚ 9.26953125 disk.sdb.in‚3.1875 disk.sda.out‚935.27880859375Ë…ª¶’ñ‚(:‡ †ÁêÔÛ±õÿ gpu.0.gpu‚3.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚165.598' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚41.3995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0& ú”ò‚(:‰ •ÁêÔÄøŒþ gpu.0.gpu‚53.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚166.207' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚41.55175 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¾ƒ¡I]ó‚(:Ò •ÁêÔÕÄÕÿ$ disk./.usageGB‚2713.121280670166 disk.sda1.in‚0 disk.dm-0.in‚ 1102.390625 disk.md0p1.in‚ 9.26953125 disk.nvme0n1.in‚ 603.58984375# disk.nvme0n1.out‚44059.74609375 disk.rbd0.out‚2.1875! disk.sdb2.out‚936.86474609375 disk.md0p1.out‚ 902.921875 disk.sda.in‚ 6.08203125! disk.sda2.out‚936.86474609375 disk.rbd0.in‚0 disk.sdb1.in‚0$ memory_percent‚91.55380725860596 disk.dm-0.out‚91727.26171875 disk.sdb.out‚936.86474609375 disk.sda.out‚936.86474609375 proc.cpu.threads‚17! disk.nvme1n1.out‚ 47667.515625 disk.md0.in‚ 9.26953125 cpu‚25.196709716927053 disk.nvme1n1.in‚ 498.80078125 disk.rbd2.in‚0 disk.rbd2.out‚2.125) proc.memory.availableMB‚ 1383.82421875 network.sent‚ 1.78282e+07 disk.sdb.in‚3.1875 disk.md0.out‚ 902.921875 disk.sda2.in‚ 6.08203125 proc.memory.rssMB‚ 2912.71875) proc.memory.percent‚17.77782440185547) disk./.usagePercent‚75.86000653701069 disk.sdb2.in‚3.1875 disk.sda1.out‚0 disk.sdb1.out‚0 network.recv‚ 2.652131e+06,ºy*2ô- _wandbruntime‚8175  _runtime‚8175qýÇõÁ & lr-AdamW/pg2‚0.0003794942801996245 trainer/global_step‚7049 % _timestamp‚1.7885185501370578e+09 _runtime‚8175.607526978 _step‚575 & lr-AdamW/pg1‚0.0003794942801996245otRýÌöÆ _runtime‚8175.607526978 _step‚575 & lr-AdamW/pg1‚0.0003794942801996245 & lr-AdamW/pg2‚0.0003794942801996245 trainer/global_step‚7049 % _timestamp‚1.7885185501370578e+09¿XÍà`2÷- _wandbruntime‚8175  _runtime‚8175P*”¡û'øõO YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚586295776596480 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1187 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚573430804113768 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 321705884 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8000919833600 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚578293939352064 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 176628478 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410816 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚647187413074944 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚578293957355008 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 _step‚576 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 176602390 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8001819241472 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 242314240 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 145103496 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 321731974 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 399863808 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7971087680508 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚565458801982480 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 _runtime‚8175.608915884 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚586295776596480 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 132374143 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚90230453 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8353396997120 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26090 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚586294859185664 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚622 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214665 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26088 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8001819241472 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177047040 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚586294859185664 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚655541028446208 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8353615371264 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚573429889662988 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚578293939352064 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 145103494 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1212 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 trainer/global_step‚7049 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 218374144 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8000919833600ÈK8€ùúO OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1187 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚573430804113768 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 321705884 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8000919833600 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚578293939352064 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 176628478 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytpÙBÛ€$es.all.current‚ 917410816 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚647187413074944 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚578293957355008 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 _step‚576 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 176602390 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8001819241472 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 242314240 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 145103496 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 321731974 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 399863808 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7971087680508 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚565458801982480 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 _runtime‚8175.608915884 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚586295776596480 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 132374143 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚90230453 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8353396997120 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26090 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚586294859185664 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚622 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214665 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26088 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8001819241472 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177047040 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚586294859185664 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚655541028446208 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8353615371264 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚573429889662988 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚578293939352064 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 145103494 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1212 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 trainer/global_step‚7049 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 218374144 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8000919833600 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚586295776596480 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35Àª% (2ú- _wandbruntime‚8176  _runtime‚8176Bp¯Ê'û€N ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31686 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚42148766 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 145123392 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 132392213 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7973093753454 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 321776489 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚655631512510976 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚647276708355584 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007241 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚618 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚565536782076608 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚586376519708672 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899421184 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚42149382 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 230943744 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8002932058112 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 399863808 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 176653095 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 176653095 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 176626981 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚573508978418434 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26114 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7972196341826 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8002932058112 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 132391595 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1187 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 321750373 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177047040 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 176626981 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 321776489 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26114 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899421184 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚578373587650560 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 _runtime‚8176.573953006 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607©}NÆ 'ü…N KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955591168 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8354549271552 _step‚577 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917424128 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚42148766 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 145123392 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 132392213 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7973093753454 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 321776489 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚655631512510976 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚647276708355584 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007241 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚618 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚565536782076608 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚586376519708672 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899421184 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚42149382 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 230943744 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8002932058112 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 399863808 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 176653095 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 176653095 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 176626981 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚573508978418434 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26114 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7972196341826 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8002932058112 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 132391595 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1187 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 321750373 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177047040 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 176626981 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 321776489 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26114 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899421184 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚578373587650560 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 _runtime‚8176.573953006 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31686 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚634 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8411572984320 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8057473821696 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 133281393 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚590332028543488 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚582275454129664 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚590332028543488 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚42432772 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 146098394 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 146098396 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 trainer/global_step‚7099 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚651653350171648 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26088 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 177835292 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚569357823728240 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8056574413824 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 _step‚580 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8026534337616 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚590332945954304 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 146098396 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26088 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007241 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚90849255 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚632 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1187 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 399863808 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 220471296 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214665 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚569357806688880 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 323933686 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8411793455616 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410816 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 323959776 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1212 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚577385255477276 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171vÍ¢Ê(‹ úO VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26088 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 177835292 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚569357823728240 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8056574413824 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 _step‚580 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8026534337616 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚590332945954304 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 146098396 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26088 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007241 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚90849255 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚632 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1187 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 399863808 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 220471296 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214665 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚569357806688880 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 323933686 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8411793455616 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410816 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 323959776 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1212 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚577385255477276 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚634 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8411572984320 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8057473821696 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 133281393 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚590332028543488 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚582275454129664 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚590332028543488 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚42432772 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 146098394 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 146098396 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 trainer/global_step‚7099 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚651653350171648 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0ÄöFæ2Œ - _wandbruntime‚8233  _runtime‚8233MÐHÚ ÿM RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 177859883 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚590413689066496 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 323978175 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8412753723904 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899421184 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚590412771642368 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955591168 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007241 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917424128 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 177859883 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26114 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚42438481 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚651742693332480 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450988 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 133299478 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚569435803822368 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 146118294 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚590413689066496 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚582355084425216 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 399863808 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 324004291 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8057687217152 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚577463429781942 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 177885997 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177047040 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚632 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 323978175 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚90861631 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚577464344232930 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411628 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 324004291 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8028540410562 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26114 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31686 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚42437849 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 146118292 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8058586638336 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚634 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1187 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 238106624 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚651742669392384/p€… 'Ž „N OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 177885997 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177047040 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚632 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 323978175 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚90861631 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚577464344232930 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411628 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 324004291 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8028540410562 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26114 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31686 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚42437849 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 146118292 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8058586638336 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚634 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1187 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 238106624 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚651742669392384 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 177859883 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚590413689066496 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 323978175 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8412753723904 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899421184 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚590412771642368 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955591168 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007241 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917424128 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 177859883 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26114 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚42438481 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚651742693332480 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450988 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 133299478 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚569435803822368 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 146118294 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚590413689066496 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚582355084425216 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 399863808 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 324004291 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8057687217152 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚577463429781942Å‚ö߬2 - _wandbruntime‚8233  _runtime‚8233ðˆ]\ Ö /loss/FRt1/assignment_loss‚3.011962928201841 /loss/FRt2/assignment_loss‚1.576961727852517 /loss/FRt3/detection_loss‚2.1788231069627018 /loss/FBt4/detection_loss‚2.6744396924972533 2loss/SRqqt3/assignment_loss‚1.5599203384180944 _step‚582 0loss/FRt3/assignment_loss‚1.8306330921857252 1loss/FBt3/assignment_loss‚0.24973136584630992 1loss/FBt4/assignment_loss‚0.10327892545519717 /loss/FRt1/detection_loss‚2.1042596452066986 1loss/SRqqt3/detection_loss‚1.4198405051063145 2loss/SRqqt2/assignment_loss‚0.9291467750977193 ,loss/FRt4/detection_loss‚3.2040629576553 1loss/SRqqt1/detection_loss‚1.8339598262682557 /loss/FBt1/detection_loss‚3.2352379709482193 1loss/FBt1/assignment_loss‚0.17197510305559263 1loss/FBt2/assignment_loss‚0.11306474541626382 /loss/FBt2/detection_loss‚0.5663200075117251 /loss/FRt4/assignment_loss‚2.417871345173229 1loss/SRqqt1/assignment_loss‚1.295942593133077 1loss/SRqqt2/detection_loss‚0.8173530410975218 /loss/FBt3/detection_loss‚2.5786996126174926 epoch‚5 trainer/global_step‚7099 0loss/SRqqt4/detection_loss‚2.516605395823717 % _timestamp‚1.7885186077790241e+09 _runtime‚8233.249398031 2loss/SRqqt4/assignment_loss‚1.6003144774585962 /loss/FRt2/detection_loss‚1.7180260950897603 &loss/total_loss‚39.708431274079125²¶\ša‘ Û 0loss/FRt3/assignment_loss‚1.8306330921857252 1loss/FBt3/assignment_loss‚0.24973136584630992 1loss/FBt4/assignment_loss‚0.10327892545519717 /loss/FRt1/detection_loss‚2.1042596452066986 1loss/SRqqt3/detection_loss‚1.4198405051063145 2loss/SRqqt2/assignment_loss‚0.9291467750977193 ,loss/FRt4/detection_loss‚3.2040629576553 1loss/SRqqt1/detection_loss‚1.8339598262682557 /loss/FBt1/detection_loss‚3.2352379709482193 1loss/FBt1/assignment_loss‚0.17197510305559263 1loss/FBt2/assignment_loss‚0.11306474541626382 /loss/FBt2/detection_loss‚0.5663200075117251 /loss/FRt4/assignment_loss‚2.417871345173229 1loss/SRqqt1/assignment_loss‚1.295942593133077 1loss/SRqqt2/detection_loss‚0.8173530410975218 /loss/FBt3/detection_loss‚2.5786996126174926 epoch‚5 trainer/global_step‚7099 0loss/SRqqt4/detection_loss‚2.516605395823717 % _timestamp‚1.7885186077790241e+09 _runtime‚8233.249398031 2loss/SRqqt4/assignment_loss‚1.6003144774585962 /loss/FRt2/detection_loss‚1.7180260950897603 &loss/total_loss‚39.708431274079125 /loss/FRt1/assignment_loss‚3.011962928201841 /loss/FRt2/assignment_loss‚1.576961727852517 /loss/FRt3/detection_loss‚2.1788231069627018 /loss/FBt4/detection_loss‚2.6744396924972533 2loss/SRqqt3/assignment_loss‚1.5599203384180944 _step‚582Æ–*“”“’ ‚(:ˆ ÑÁêÔÝôîÿ gpu.0.gpu‚30.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚37.0 gpu.0.powerWatts‚133.958' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚33.4895 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0P!¯\“ ‚(:Ñ ÑÁêÔäÙÌ€( proc.memory.availableMB‚ 1383.6484375 disk.rbd0.in‚0 disk.nvme1n1.in‚ 498.80078125 disk.md0.out‚ 909.21875 disk.dm-0.out‚92346.82421875 network.recv‚ 2.670181e+06 disk.sda2.in‚ 6.08203125 disk.sdb1.out‚0! disk.sdb2.out‚943.39892578125 cpu‚25.19551701172707 disk.rbd0.out‚2.1875# disk.nvme1n1.out‚47668.47265625 disk.sdb2.in‚3.1875 disk.nvme0n1.in‚ 603.58984375$ memory_percent‚91.55488014221191 disk.dm-0.in‚ 1102.390625 proc.memory.rssMB‚ 2912.71875$ disk./.usageGB‚2713.870922088623 disk.sdb.in‚3.1875 disk.sda.out‚943.39892578125 disk.sda1.in‚0 disk.sda1.out‚0) proc.memory.percent‚17.77782440185547 proc.cpu.threads‚17) disk./.usagePercent‚75.88096682482006! disk.sda2.out‚943.39892578125 disk.rbd2.in‚0 network.sent‚ 1.7953492e+07 disk.sdb1.in‚0 disk.md0p1.in‚ 9.26953125 disk.md0p1.out‚ 909.21875" disk.nvme0n1.out‚ 44678.3515625 disk.sdb.out‚943.39892578125 disk.sda.in‚ 6.08203125 disk.md0.in‚ 9.26953125 disk.rbd2.out‚2.125'a¬ˆ”” ‚(:‰ àÁêÔåªïý gpu.0.gpu‚43.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚161.587' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚40.39675 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ŽnÎÜj• ‚(:ß àÁêÔÿϽÿ disk.sda1.in‚0# disk.nvme0n1.out‚44819.63671875) proc.memory.percent‚17.77782440185547( disk./.usagePercent‚75.8809659715358$ disk./.usageGB‚2713.870891571045 disk.md0p1.out‚ 911.1796875 disk.sda2.in‚ 6.08203125 disk.rbd0.in‚0 disk.sda.in‚ 6.08203125 disk.rbd0.out‚ 2.19921875# disk.nvme1n1.out‚47668.66796875 cpu‚25.195230225203964 proc.cpu.threads‚17 disk.rbd2.in‚0 disk.sdb.out‚945.42724609375 disk.md0.in‚ 9.26953125) proc.memory.availableMB‚ 1383.73828125 disk.dm-0.in‚ 1102.390625 disk.md0p1.in‚ 9.26953125 disk.nvme0n1.in‚ 603.58984375! disk.sda2.out‚945.42724609375 disk.md0.out‚ 911.1796875 network.recv‚ 2.675834e+06 disk.sdb.in‚3.1875 disk.nvme1n1.in‚ 498.80078125 disk.sdb1.out‚0 proc.memory.rssMB‚ 2912.71875 disk.sda1.out‚0 disk.dm-0.out‚ 92488.3046875 disk.sda.out‚945.42724609375 disk.sdb1.in‚0$ memory_percent‚91.55433177947998 network.sent‚ 1.8061417e+07 disk.rbd2.out‚ 2.19921875 disk.sdb2.in‚3.1875! disk.sdb2.out‚945.42724609375X”– ‚(:‰ ïÁêÔÖúüý gpu.0.gpu‚12.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚37.0 gpu.0.powerWatts‚162.649' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚40.66225 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¿ë:]— ‚(:Ò ïÁêÔŽŠÿ disk.md0p1.in‚ 9.26953125 network.sent‚ 1.8067203e+07 disk.sda1.out‚0 disk.md0p1.out‚ 912.61328125 disk.sdb.in‚3.1875) proc.memory.availableMB‚ 1383.60546875 proc.memory.rssMB‚ 2912.71875 cpu‚25.194939836158817$ disk./.usageGB‚2713.870948791504 disk.sda.out‚ 946.9140625 disk.nvme1n1.in‚ 498.80078125 disk.md0.in‚ 9.26953125 disk.md0.out‚ 912.61328125 disk.sdb2.out‚ 946.9140625 disk.dm-0.in‚ 1102.390625 disk.sda.in‚ 6.08203125 disk.rbd0.out‚ 2.26171875# disk.nvme1n1.out‚47668.97265625$ memory_percent‚91.55514240264893) proc.memory.percent‚17.77782440185547 network.recv‚ 2.679807e+06 disk.sdb1.in‚0 disk.sdb2.in‚3.1875 disk.sda1.in‚0 disk.sdb.out‚ 946.9140625 disk.sda2.in‚ 6.08203125 disk.dm-0.out‚92770.30859375 disk.nvme0n1.in‚ 603.58984375" disk.nvme0n1.out‚ 45101.3359375) disk./.usagePercent‚75.88096757144378 disk.rbd2.out‚ 2.19921875 disk.sda2.out‚ 946.9140625 disk.rbd0.in‚0 disk.sdb1.out‚0 disk.rbd2.in‚0 proc.cpu.threads‚17 >Z“˜ ‚(:ˆ þÁêÔ¦ô‘þ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚36.0 gpu.0.powerWatts‚114.131' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.53275 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ã&£Ãd™ ‚(:Ù þÁêÔÏÿªÿ disk.md0p1.out‚ 913.94140625 disk.sda2.out‚ 948.30859375$ memory_percent‚91.55738353729248 proc.cpu.threads‚17 disk.sdb2.in‚3.1875 disk.sdb2.out‚ 948.30859375 network.recv‚ 2.684449e+06 disk.sdb1.in‚0 disk.md0.out‚ 913.94140625 cpu‚25.194676404348343 disk.sda.out‚ 948.30859375 disk.md0.in‚ 9.26953125 disk.sda1.in‚0 disk.sdb.in‚3.1875# disk.nvme1n1.out‚47669.17578125 disk.sdb1.out‚0 disk.sda1.out‚0 disk.dm-0.out‚ 92842.3984375 disk.md0p1.in‚ 9.26953125# disk.nvme0n1.out‚45173.22265625) proc.memory.availableMB‚ 1383.23828125 disk.rbd2.out‚ 2.30126953125 disk.sdb.out‚ 948.30859375 disk.sda.in‚ 6.08203125 disk.rbd0.in‚0 disk.rbd0.out‚ 2.26171875 disk.rbd2.in‚0 network.sent‚ 1.8073622e+07) disk./.usagePercent‚75.88096607819634 disk.dm-0.in‚ 1102.390625 disk.nvme0n1.in‚ 603.58984375 disk.sda2.in‚ 6.08203125 disk.nvme1n1.in‚ 498.80078125 proc.memory.rssMB‚ 2912.71875) proc.memory.percent‚17.77782440185547$ disk./.usageGB‚2713.870895385742¨á2š - _wandbruntime‚8289  _runtime‚8289O†UWÉ› à trainer/global_step‚7149 % _timestamp‚1.7885186639052923e+09 _runtime‚8289.375763119 _step‚583 ' lr-AdamW/pg1‚0.00036721694949453233 ' lr-AdamW/pg2‚0.00036721694949453233zšÑdÎœ È % _timestamp‚1.7885186639052923e+09 _runtime‚8289.375763119 _step‚583 ' lr-AdamW/pg1‚0.00036721694949453233 ' lr-AdamW/pg2‚0.00036721694949453233 trainer/global_step‚7149Çæiyû2 - _wandbruntime‚8289  _runtime‚8289”Lá—û'ž õO ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8113128401920 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚594369197901312 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚573256828434640 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 326187578 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8082878406144 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 _runtime‚8289.376960444 _step‚584 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955591168 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚42721946 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚656119335148544 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 trainer/global_step‚7149 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917410816 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407872 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚586256986910208 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26088 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8113128401920 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 179094282 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1187 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚664589155448320 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407872 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214665 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177047040 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 147093296 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚91468055 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26088 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411420 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚639 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚91468053 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 179068194 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚594370115312128 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8112228994048 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚586256986910208 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚594370115312128 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 179068194 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 134190640 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 134189999 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚664589397762560 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚586256968907264 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410816 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 399863808 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚656119311208448 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 218374144 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450780 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8081980994724 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 326161488 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 147093294 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚641 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚573256811395280 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1212 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0ÞRbŒŸ úO UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚594369197901312 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 242314240 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚586256986910208 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚594370115312128 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 179068194 RDDeviceStatsMonitor.on_train_batch_start}M7ªt%/inactive_split.all.allocated‚ 134190640 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 134189999 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚664589397762560 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚586256968907264 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410816 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 399863808 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚656119311208448 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 218374144 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450780 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8081980994724 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 326161488 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 147093294 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚641 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚573256811395280 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1212 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8113128401920 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚594369197901312 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚573256828434640 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 326187578 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8082878406144 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 _runtime‚8289.376960444 _step‚584 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955591168 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚42721946 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚656119335148544 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 trainer/global_step‚7149 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917410816 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407872 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚586256986910208 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26088 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8113128401920 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 179094282 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1187 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚664589155448320 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407872 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214665 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177047040 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 147093296 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚91468055 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26088 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411420 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚639 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚91468053 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 179068194 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚594370115312128 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8112228994048 JDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 326232093 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 147113192 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31686 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚42728211 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 147113194 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 399863808 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8471228576256 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899421184 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 326232093 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 179092785 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8083089656042 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899421184 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917424128 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚609 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1187 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 179092785 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚586336617205760 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 147113194 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955591168 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177047040 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 222555136 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚594450858424320 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚611 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 134208031 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607‘™|% '¢ „N PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 326232093 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 147113192 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31686 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚42728211 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 147113194 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 399863808 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8471228576256 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899421184 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 326232093 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 179092785 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8083089656042 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899421184 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917424128 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚609 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1187 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 179092785 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚586336617205760 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 147113194 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955591168 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177047040 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 222555136 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚594450858424320 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚611 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 134208031 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚586336617205760 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8471006021120 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚581417881145450 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚594450858424320 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 134208642 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8083987067670 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8113341797376 FDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _step‚588 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚598407284669952 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8528314977280 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚585294158204292 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚577155833141040 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8137427651832 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 328389290 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚598406367259136 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407872 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411420 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 180301096 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚590238501687808 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚598406367259136 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚616 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 _runtime‚8342.834352212 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8167883574272 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚669113829536768 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407872 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26090 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8167883574272 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚92086853 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214665 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450780 trainer/global_step‚7199 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚660585296185344 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚616 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 _runtime‚8342.834352212 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8167883574272 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚669113829536768 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407872 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26090 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8167883574272 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚92086853 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214665 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450780 trainer/global_step‚7199 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚660585296185344 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _step‚588 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚598407284669952 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8528314977280 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚585294158204292 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚577155833141040 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8137427651832 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 328389290 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚598406367259136 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407872 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411420 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 180301096 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚590238501687808 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚598406367259136 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411628 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚598487110358016 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚590318131983360 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 148108094 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31686 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚577233813235168 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 328433779 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955591168 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8168996377600 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 L>DeviceStatsMonitor.on_train_ba¹W…—ztch_end/allocation.all.allocated‚ 328459895 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚92099229 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8139433724778 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚92099231 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚43017146 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚585373246959946 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007241 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 180351801 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1187 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8168996377600 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177047040 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8529692306432 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450988 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 328459895 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚598488027782144 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 135116375 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8138536313150 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 135117006 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚631 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚43017775 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚669204307712512 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 259078144Ä-'Š '¶ …N S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚598487110358016 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚590318131983360 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 148108094 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31686 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚577233813235168 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 328433779 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955591168 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8168996377600 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 328459895 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚92099229 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8139433724778 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚92099231 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚43017146 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚585373246959946 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007241 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 180351801 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1187 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8168996377600 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177047040 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8529692306432 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450988 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 328459895 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚598488027782144 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 135116375 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8138536313150 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 135117006 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚631 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚43017775 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚669204307712512 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 259078144 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚669204048634368 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚8343.791324363 _step‚589 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 148108094 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8529457168384 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 148108092 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26114 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917424128 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚590318131983360 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 180351801 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411628Í2çÅO2· - _wandbruntime‚8343  _runtime‚8343D îÙ\¸ Ö 2loss/SRqqt4/assignment_loss‚1.5545582169246288 1loss/FBt3/assignment_loss‚0.04217990927069574 /loss/FRt2/detection_loss‚1.9970814707533766 0loss/SRqqt1/detection_loss‚2.050165542616294 1loss/SRqqt4/detection_loss‚2.3942281507676646 epoch‚5 2loss/SRqqt1/assignment_loss‚1.5257472273869774 /loss/FBt3/detection_loss‚2.9525455137093863 trainer/global_step‚7199 /loss/FRt3/assignment_loss‚2.217665177688264 0loss/FRt4/assignment_loss‚2.9810554958306827 2loss/SRqqt2/assignment_loss‚0.9422649114762676 0loss/FBt4/assignment_loss‚0.3394509552260085 /loss/FRt3/detection_loss‚1.9426195497314136 /loss/FBt1/detection_loss‚2.0344518025716147 0loss/FRt1/assignment_loss‚2.6518865411098185 1loss/FBt2/assignment_loss‚0.10934812745863934 .loss/FRt1/detection_loss‚2.322056034459448 /loss/FRt4/detection_loss‚3.1284591211722446 1loss/SRqqt2/detection_loss‚0.9771530416424143 /loss/FBt2/detection_loss‚0.6034591439358123 .loss/FBt4/detection_loss‚2.116625482837359 _runtime‚ 8343.79293318 2loss/SRqqt3/assignment_loss‚0.8767129686035093 0loss/SRqqt3/detection_loss‚1.386372680302526 % _timestamp‚1.7885187183227935e+09 0loss/FRt2/assignment_loss‚1.4683223304649193 1loss/FBt1/assignment_loss‚0.14055669135026014 %loss/total_loss‚38.75496608729023 _step‚590—Ââ…a¹ Û 0loss/FRt2/assignment_loss‚1.4683223304649193 1loss/FBt1/assignment_loss‚0.14055669135026014 %loss/total_loss‚38.75496608729023 _step‚590 2loss/SRqqt4/assignment_loss‚1.5545582169246288 1loss/FBt3/assignment_loss‚0.04217990927069574 /loss/FRt2/detection_loss‚1.9970814707533766 0loss/SRqqt1/detection_loss‚2.050165542616294 1loss/SRqqt4/detection_loss‚2.3942281507676646 epoch‚5 2loss/SRqqt1/assignment_loss‚1.5257472273869774 /loss/FBt3/detection_loss‚2.9525455137093863 trainer/global_step‚7199 /loss/FRt3/assignment_loss‚2.217665177688264 0loss/FRt4/assignment_loss‚2.9810554958306827 2loss/SRqqt2/assignment_loss‚0.9422649114762676 0loss/FBt4/assignment_loss‚0.3394509552260085 /loss/FRt3/detection_loss‚1.9426195497314136 /loss/FBt1/detection_loss‚2.0344518025716147 0loss/FRt1/assignment_loss‚2.6518865411098185 1loss/FBt2/assignment_loss‚0.10934812745863934 .loss/FRt1/detection_loss‚2.322056034459448 /loss/FRt4/detection_loss‚3.1284591211722446 1loss/SRqqt2/detection_loss‚0.9771530416424143 /loss/FBt2/detection_loss‚0.6034591439358123 .loss/FBt4/detection_loss‚2.116625482837359 _runtime‚ 8343.79293318 2loss/SRqqt3/assignment_loss‚0.8767129686035093 0loss/SRqqt3/detection_loss‚1.386372680302526 % _timestamp‚1.7885187183227935e+09ÎÕŸ·­“º ‚(:ˆ ÉÂêÔñª÷ý gpu.0.gpu‚41.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚172.118' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.0295 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ¬>Ño» ‚(:ä ÉÂêÔ»º”ÿ disk.md0p1.in‚ 9.26953125 disk.nvme0n1.in‚ 603.58984375! disk.sdb2.out‚957.05126953125 disk.dm-0.in‚ 1102.390625 disk.sdb.out‚957.05126953125 disk.rbd0.out‚ 2.466796875 disk.sdb1.out‚0) disk./.usagePercent‚75.85148628038375 disk.sdb2.in‚3.1875 disk.rbd0.in‚0 disk.md0.out‚ 922.39453125) proc.memory.percent‚17.77782440185547 cpu‚25.193203744505812 disk.sdb.in‚3.1875 disk.sda.in‚ 6.08203125 disk.md0.in‚ 9.26953125 disk.sda1.in‚0 disk.sda2.in‚ 6.08203125 disk.nvme1n1.in‚ 498.80078125 disk.sdb1.in‚0 disk.rbd2.out‚ 2.40380859375( proc.memory.availableMB‚ 1383.3671875 network.recv‚ 2.705201e+06% disk./.usageGB‚2712.8165550231934 disk.md0p1.out‚ 922.39453125 disk.sda1.out‚0 disk.dm-0.out‚93711.01171875 disk.sda.out‚957.05126953125# disk.nvme1n1.out‚47670.35546875 proc.memory.rssMB‚ 2912.71875 proc.cpu.threads‚17 disk.nvme0n1.out‚ 46040.65625! disk.sda2.out‚957.05126953125 disk.rbd2.in‚0$ memory_percent‚91.55659675598145 network.sent‚ 1.8203265e+07ÆÞƒ”¼ ‚(:‰ ØÂêÔ³ˆ‘þ gpu.0.gpu‚55.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚170.559' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.63975 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0kf½ ‚(:Û ØÂêÔÓñšÿ disk.sdb1.out‚0 disk.sda.out‚958.52001953125 disk.sda2.in‚ 6.08203125 disk.sdb2.in‚3.1875 disk.dm-0.out‚93712.19921875$ memory_percent‚91.55502319335938 network.sent‚ 1.8311272e+07" disk.nvme1n1.out‚ 47670.5703125 disk.rbd2.in‚0 disk.sda1.in‚0 disk.dm-0.in‚ 1102.390625 proc.cpu.threads‚17 disk.rbd0.in‚0 disk.nvme1n1.in‚ 498.80078125 disk.sda.in‚ 6.08203125 proc.memory.rssMB‚ 2912.71875) disk./.usagePercent‚75.84895031957201! disk.sda2.out‚958.52001953125 disk.sdb.out‚958.52001953125 disk.rbd0.out‚ 2.466796875 disk.md0.in‚ 9.26953125$ proc.memory.availableMB‚1383.625 cpu‚25.1929177718328# disk.nvme0n1.out‚46041.62890625 disk.sdb.in‚3.1875 network.recv‚ 2.711016e+06 disk.sdb1.in‚0 disk.md0.out‚ 923.8046875 disk.rbd2.out‚ 2.466796875 disk.md0p1.in‚ 9.26953125 disk.nvme0n1.in‚ 603.58984375$ disk./.usageGB‚2712.725856781006! disk.sdb2.out‚958.52001953125 disk.sda1.out‚0 disk.md0p1.out‚ 923.8046875) proc.memory.percent‚17.77782440185547§?ü“¾ ‚(:ˆ çÂêÔŒœãý gpu.0.gpu‚42.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚159.258' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚39.8145 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Xüˆƒl¿ ‚(:á çÂêÔüóèþ network.recv‚ 2.715139e+06 disk.rbd0.in‚0 disk.rbd0.out‚ 2.466796875 disk.md0.out‚ 925.29296875 disk.sdb.in‚3.1875 disk.sdb.out‚960.0673828125 network.sent‚ 1.8317214e+07 disk.sda2.in‚ 6.08203125 disk.sda2.out‚960.0673828125 disk.nvme1n1.in‚ 498.80078125" disk.nvme1n1.out‚ 47670.7578125 disk.sdb1.out‚0 disk.dm-0.out‚93856.43359375 disk.md0.in‚ 9.26953125 disk.rbd2.in‚0 disk.md0p1.out‚ 925.29296875 disk.nvme0n1.in‚ 603.58984375 disk.sda.out‚960.0673828125$ memory_percent‚91.55504703521729) proc.memory.availableMB‚ 1383.62109375% disk./.usageGB‚2712.8667335510254 disk.sdb1.in‚0 disk.sdb2.in‚3.1875 disk.sdb2.out‚960.0673828125 disk.sda1.in‚0# disk.nvme0n1.out‚46185.67578125 proc.memory.rssMB‚ 2912.71875 proc.cpu.threads‚17) disk./.usagePercent‚75.85288929302327 disk.sda.in‚ 6.08203125) proc.memory.percent‚17.77782440185547 disk.rbd2.out‚ 2.466796875 disk.sda1.out‚0 disk.dm-0.in‚ 1102.390625 disk.md0p1.in‚ 9.26953125 cpu‚25.192605885398415¶ëín2À - _wandbruntime‚8396  _runtime‚8396$–¨LÉÁ à ' lr-AdamW/pg2‚0.00035499099330434515 trainer/global_step‚7249 % _timestamp‚1.7885187711801457e+09 _runtime‚8396.650646007 _step‚591 ' lr-AdamW/pg1‚0.00035499099330434515’Î2ºÎ È ' lr-AdamW/pg1‚0.00035499099330434515 ' lr-AdamW/pg2‚0.00035499099330434515 trainer/global_step‚7249 % _timestamp‚1.7885187711801457e+09 _runtime‚8396.650646007 _step‚591ÏÂœt$2à - _wandbruntime‚8396  _runtime‚8396äc1ü'Ä öO WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26090 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚92705655 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 181533998 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 181560086 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 136005762 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1187 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8586571424768 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007241 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407872 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 330643182 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚43299491 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 149083094 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚594220016465408 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚594219998462464 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407872 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚618 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚673637804706816 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450780 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚594220016465408 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚665051233282048 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26088 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 % _timestamp‚1.7885187711814172e+09 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955591168 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 136005144 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955591168 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚616 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚602444454027776 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8224437562368 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26088 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8224437562368 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚594219998462464 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚602444454027776 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8586796090368 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚602443536616960 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 149083094 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚589247695117020 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177047040 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 248605696 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 _step‚592 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917410816 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8223538154496 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚581054837847440 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1212 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 181560086 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 181533998 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171ú­¬Ñ#Å ûO F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 399863808 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8193771720360 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26090 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚92705655 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 181533998 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 181560086 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 136005762 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1187 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8586571424768 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007241 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407872 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 330643182 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚43299491 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 149083094 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚594220016465408 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚594219998462464 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407872 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚618 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚673637804706816 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450780 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚594220016465408 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚665051233282048 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26088 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 % _timestamp‚1.7885187711814172e+09 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955591168 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 136005144 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955591168 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚616 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚602444454027776 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8224437562368 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26088 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8224437562368 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚594219998462464 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚602444454027776 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8586796090368 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚602443536616960 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 149083094 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚589247695117020 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177047040 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 248605696 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 _step‚592 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917410816 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8223538154496 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚581054837847440 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1212 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 181560086 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 181533998 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 330687697 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚594299646760960 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚594299628758016 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚589326783872466 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8193982970258 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 330661581 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8194880381886 _runtime‚8397.707354554 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 218360832 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450988 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 149102992 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26114 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚92718031 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚43305892 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 242300928 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚594299646760960 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 136023923 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31686 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚594299628758016 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 136023285 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚581132800902208 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917424128 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8224650957824 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 330661581 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚43305256 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177047040 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0{. 'È …N T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚602525197139968 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8225550379008 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899421184 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚589327698323454 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007241 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚636 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1187 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚602524279715840 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177047040 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411628 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 330687697 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚594299646760960 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚594299628758016 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚589326783872466 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8193982970258 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 330661581 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8194880381886 _runtime‚8397.707354554 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 218360832 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450988 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 149102992 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26114 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚92718031 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚43305892 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 242300928 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚594299646760960 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 136023923 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31686 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚594299628758016 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 136023285 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚581132800902208 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917424128 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8224650957824 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 330661581 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚43305256 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 UEDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.peak‚ 13510214665 ]NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.current‚ 1182793728 QIDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.allocated‚564 VHDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.current‚ 690184120 aMDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.freed‚583883455680096 OGDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.current‚607 OGDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.current‚564 trainer/global_step‚7253 _step‚595 VHDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.freed‚ 182091380 [MDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.current‚ 162518528 fRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.freed‚668214853513728 KEDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_validation_batch_start/active.all.allocated‚ 332043009 OGDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.current‚193 BDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 ^JDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.allocated‚592106661807126 [KDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 FDeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 UEDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.peak‚ 13510214665 ]NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.current‚ 1182793728 QIDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.allocated‚564 VHDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.current‚ 690184120 aMDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.freed‚583883455680096 OGDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.current‚607 OGDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.current‚564 trainer/global_step‚7253 _step‚595 VHDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.freed‚ 182091380 [MDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.current‚ 162518528 fRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.freed‚668214853513728 KEDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_validation_batch_start/active.all.allocated‚ 332043009 OGDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.current‚193 BDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 ^JDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.allocated‚592106661807126 [KDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.peak‚ 13037993984ÓÒýÖ 2Ù - _wandbruntime‚8461  _runtime‚8461qIzc4 Ú çR WJDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.current‚18002944 _KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.freed‚583933225769920 _runtime‚8461.546468903 ZJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 ]NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 % _timestamp‚1.7885188360756764e+09 MGDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.current‚0 LFDeviceStatsMonitor.on_validation_batch_end/oversize_segments.allocated‚0 [MDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.current‚ 675116544 ICDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.freed‚0 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 @9DeviceStatsMonitor.on_validation_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_validation_batch_end/num_sync_all_streams‚2 ZJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.peak‚ 12392107008 KADeviceStatsMonitor.on_validation_batch_end/active.small_pool.peak‚31686 QCDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.current‚ 693119488 bPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.freed‚ 8617153110016 HBDeviceStatsMonitor.on_validation_batch_end/oversize_segments.freed‚0 RHDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.current‚20519 TFDeviceStatsMonitor.on_validation_batch_end/active.small_pool.allocated‚ 182120673 VHDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.peak‚ 399863808 @:DeviceStatsMonitor.on_validation_batch_end/num_device_free‚0 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.freed‚0 I;DeviceStatsMonitor.on_validation_batch_end/active.all.freed‚ 332046538 JADeviceStatsMonitor.on_validation_batch_end/active.large_pool.peak‚3677 `RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.current‚ 138578432 TFDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.current‚ 693119488 ZMDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.current‚18002944 NHDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.current‚2 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.allocated‚93239535 PIDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.peak‚35 RBDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.peak‚ 14220787712 M?DeviceStatsMonitor.on_validation_batch_end/active.all.allocated‚ 332067059 XDDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.freed‚605550481492480 JDDeviceStatsMonitor.on_validation_batch_end/oversize_segments.current‚0 [MDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.current‚ 673144760 ]KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.freed‚ 8253767763456 XDDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.freed‚59215622863955LHå›94 aODeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.allocated‚ 8223676014394 \LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.current‚ 13037993984 ICDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.freed‚0 _step‚596 TFDeviceStatsMonitor.on_validation_batch_end/active.large_pool.allocated‚ 149946386 _KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.freed‚597296713729024 OEDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.peak‚31686 D;DeviceStatsMonitor.on_validation_batch_end/num_device_alloc‚1171 \HDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.freed‚597296713729024 dPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.freed‚668270302985728 XJDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.current‚ 675116544 ZHDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.freed‚ 8253767763456 aMDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.allocated‚676887618614272 KEDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.freed‚0 TFDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.current‚ 690184120 YJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.peak‚ 1175007241 OGDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.allocated‚607 MEDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.current‚607 JDDeviceStatsMonitor.on_validation_batch_end/active.large_pool.current‚2 \HDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.allocated‚605551174611968 cODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.allocated‚597296731731968 QCDeviceStatsMonitor.on_validation_batch_end/allocation.all.allocated‚ 332067059 KADeviceStatsMonitor.on_validation_batch_end/allocation.all.current‚20521 H>DeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚31931 fTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.allocated‚ 8617291688448 cODeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.allocated‚583933242809280 ^LDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.allocated‚ 8254442880000 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 C=DeviceStatsMonitor.on_validation_batch_end/num_oom_rejections‚0 WGDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.allocated‚ 14220787712 G>DeviceStatsMonitor.on_validation_batch_end/segment.all.current‚1171 OGDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.allocated‚564 UGDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.allocated‚ 136849509 GADeviceStatsMonitor.on_validation_batch_end/oversize_segments.peak‚0 ZMDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.current‚17039360 YIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 JBDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.peak‚607 MEDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.current‚564 ICDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.freed‚0 XJDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.allocated‚ 182120673 TFDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.freed‚ 182100154 hTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.allocated‚668270326925824 JDDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 I@DeviceStatsMonitor.on_validation_batch_end/segment.all.allocated‚1171 G=DeviceStatsMonitor.on_validation_batch_end/active.all.current‚20521 NDDeviceStatsMonitor.on_validation_batch_end/active.small_pool.current‚20519 WGDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.peak‚ 12835443712 KBDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.peak‚1212 YKDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.current‚ 162518528 OIDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.allocated‚0 ^NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 D;DeviceStatsMonitor.on_validation_batch_end/segment.all.peak‚1171 MEDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.current‚193 P@DeviceStatsMonitor.on_validation_batch_end/active_bytes.all.peak‚ 13955591168 RIDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.peak‚1187 93DeviceStatsMonitor.on_validation_batch_end/num_ooms‚0 UEDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.current‚ 14220787712 PBDeviceStatsMonitor.on_validation_batch_end/active.large_pool.freed‚ 149946384 `LDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.allocated‚597296731731968 QCDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.freed‚ 136849316 _RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.current‚23940096 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.freed‚0 M?DeviceStatsMonitor.on_validation_batch_end/allocation.all.freed‚ 332046538 RLDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.current‚2 JBDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.peak‚564 TLDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.current‚191 [LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.current‚ 1182793728 PBDeviceStatsMonitor.on_validation_batch_end/active.small_pool.freed‚ 182100154 UADeviceStatsMonitor.on_validation_batch_end/active_bytes.all.freed‚605550481492480 XJDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.allocated‚ 149946386 SCDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.peak‚ 13510214665 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.freed‚93239533 BDeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚31931 fTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.allocated‚ 8617291688448 cODeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.allocated‚583933242809280 ^LDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.allocated‚ 8254442880000 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 C=DeviceStatsMonitor.on_validation_batch_end/num_oom_rejections‚0 WGDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.allocated‚ 14220787712 G>DeviceStatsMonitor.on_validation_batch_end/segment.all.current‚1171 OGDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.allocated‚564 UGDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.allocated‚ 136849509 GADeviceStatsMonitor.on_validation_batch_end/oversize_segments.peak‚0 ZMDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.current‚17039360 YIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 JBDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.peak‚607 MEDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.current‚564 ICDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.freed‚0 XJDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.allocated‚ 182120673 TFDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.freed‚ 182100154 hTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.allocated‚668270326925824 JDDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 I@DeviceStatsMonitor.on_validation_batch_end/segment.all.allocated‚1171 G=DeviceStatsMonitor.on_validation_batch_end/active.all.current‚20521 NDDeviceStatsMonitor.on_validation_batch_end/active.small_pool.current‚20519 WGDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.peak‚ 12835443712 KBDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.peak‚1212 YKDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.current‚ 162518528 OIDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.allocated‚0 ^NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 D;DeviceStatsMonitor.on_validation_batch_end/segment.all.peak‚1171 MEDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.current‚193 P@DeviceStatsMonitor.on_validation_batch_end/active_bytes.all.peak‚ 13955591168 RIDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.peak‚1187 93DeviceStatsMonitor.on_validation_batch_end/num_ooms‚0 UEDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.current‚ 14220787712 PBDeviceStatsMonitor.on_validation_batch_end/active.large_pool.freed‚ 149946384 `LDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.allocated‚597296731731968 QCDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.freed‚ 136849316 _RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.current‚23940096 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.freed‚0 M?DeviceStatsMonitor.on_validation_batch_end/allocation.all.freed‚ 332046538 RLDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.current‚2 JBDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.peak‚564 TLDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.current‚191 [LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.current‚ 1182793728 PBDeviceStatsMonitor.on_validation_batch_end/active.small_pool.freed‚ 182100154 UADeviceStatsMonitor.on_validation_batch_end/active_bytes.all.freed‚605550481492480 XJDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.allocated‚ 149946386 SCDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.peak‚ 13510214665 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.freed‚93239533 B&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 :"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.35344988107681274 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 jet/accuracy_6_of_6‚0 6 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.326577752828598 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 :"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.42467254400253296 :"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.37919387221336365 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.8571428656578064 )loss/val_total_loss‚39.21357400732002 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.38713526725769043 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 9"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.3297979533672333 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 :"Purity/*FRt2SRqqt1FBt/event_purity‚0.02241748757660389 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 ,"Purity/0FRt0SRqqt3FBt/event_purity‚0.125 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.9444444179534912 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.3541666567325592 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 ("Purity/*FRt2SRqqt3FBt/SRqqt_purity‚1 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 9"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.4000000059604645 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 8"Purity/0FRt*SRqqt2FBt/event_purity‚0.279652863740921 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 * Purity/1FRt0SRqqt3FBt/FRt_purity‚0.625 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.3238120973110199 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 8"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.618055522441864 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9977622032165527 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9974977374076843 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.3369171917438507 *jet/accuracy_1_of_4‚0.9094056487083435 & Purity/*FRt3SRqqt1FBt/FBt_purity‚1 +"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.25 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.2181648313999176 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.014971093274652958 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.2777777910232544 jet/accuracy_5_of_7‚0 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.11365262418985367 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.6388888955116272 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.5415721535682678 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 5 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.46123206615448 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.3010818362236023 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 particle/accuracy_1_of_3‚1 +jet/accuracy_3_of_6‚0.17307692766189575 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9700000286102295 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9893267154693604 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9901393055915833 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 )jet/accuracy_1_of_2‚0.733317494392395 & Purity/*FRt3SRqqt*FBt/FBt_purity‚1 jet/accuracy_4_of_7‚0 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.5417117476463318 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9952331185340881 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.32894736528396606 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.3363457918167114 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.3402777910232544 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9735663533210754 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.4253290295600891 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.7727272510528564 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9913323521614075 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.3333333134651184 particle/accuracy_1_of_6‚1 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.4642857015132904 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 ."Purity/1FRt0SRqqt3FBt/event_purity‚0.34375 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 6 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.301118940114975 particle/accuracy_2_of_4‚1 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 :"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.36772024631500244 8 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.32777780294418335 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.1291113793849945 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5534499883651733 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 7 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.3263888955116272 ("Purity/4FRt*SRqqt*FBt/SRqqt_purity‚1 :"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.37348467111587524 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9882693290710449 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.4174850881099701 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 particle/accuracy_1_of_7‚1 & Purity/*FRt*SRqqt4FBt/FRt_purity‚0 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 :"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.35643884539604187 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.31101956963539124 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9942528009414673 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 particle/accuracy_2_of_7‚1 & Purity/0FRt3SRqqt1FBt/FBt_purity‚1 7 Purity/0FRt2SRqqt1FBt/FBt_purity‚0.9811320900917053 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9079861044883728 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.3333333432674408 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.5000000596046448 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9832080006599426 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9896175265312195 7 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.2776707112789154 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 :"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.41604891419410706 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9766082167625427 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2688491940498352 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9179894328117371 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.17554017901420593 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.5000000596046448 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 jet/accuracy_6_of_7‚0 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.29579952359199524 <"Purity/1FRt1SRqqt2FBt/event_purity‚0.0069444444961845875 6 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.986111044883728 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 jet/accuracy_5_of_6‚0 & Purity/0FRt3SRqqt*FBt/FBt_purity‚1 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 ) Purity/4FRt1SRqqt0FBt/FRt_purity‚0.25 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 (particle/f_score‚0.42521730065345764 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.5115740895271301 +"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.25 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.4529609680175781 :"Purity/*FRt0SRqqt2FBt/event_purity‚0.28404977917671204 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.23822280764579773 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9993098378181458 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.00607638899236Ç{-‡`9175 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.3677172362804413 7 Purity/3FRt1SRqqt2FBt/FRt_purity‚0.3333333432674408 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.5106133818626404 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.3576986491680145 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.03900284320116043 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 7 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.3065843880176544 8"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.675438642501831 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.3333333432674408 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.3380413353443146 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.5515351295471191 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.3414141535758972 ,"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.875 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚1 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6000719666481018 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.07894736528396606 :"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.46666666865348816 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.2944486439228058 7 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.6176470518112183 5 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.32005575299263 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.32138946652412415 7 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.2953703999519348 7 Purity/3FRt4SRqqt*FBt/FRt_purity‚0.3333333432674408 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9951940178871155 + Purity/0FRt0SRqqt4FBt/FBt_purity‚0.8125 , Purity/*FRt2SRqqt2FBt/FRt_purity‚0.28125 jet/accuracy_3_of_7‚0 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 particle/accuracy_3_of_5‚1 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.04428185895085335 9"Purity/2FRt*SRqqt*FBt/event_purity‚0.0155824339017272 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 *jet/accuracy_1_of_5‚0.9473630785942078 :"Purity/0FRt1SRqqt0FBt/event_purity‚0.35344988107681274 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.3836187720298767 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 particle/accuracy_4_of_5‚1 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8500000238418579 +"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.25 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.05031375586986542 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 6 Purity/2FRt*SRqqt1FBt/FBt_purity‚0.998236358165741 particle/accuracy_2_of_5‚1 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.4340277910232544 :"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.40740740299224854 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.7035030126571655 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 ;"Purity/0FRt2SRqqt0FBt/event_purity‚0.046031802892684937 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.1187903881072998 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 8"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.459770143032074 ;"Purity/0FRt3SRqqt0FBt/event_purity‚0.026143791154026985 7 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.2993900179862976 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8197475075721741 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.3214285671710968 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 ) Purity/4FRt1SRqqt*FBt/FRt_purity‚0.25 8 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.48517507314682007 8 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.42111989855766296 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 jet/accuracy_5_of_5‚0 particle/accuracy_4_of_4‚1 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.3645833432674408 *"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.5 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.5430143475532532 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 particle/accuracy_2_of_2‚1 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 & Purity/*FRt0SRqqt4FBt/FRt_purity‚0 & Purity/*FRt3SRqqt2FBt/FRt_purity‚0 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.5922618508338928 :"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.40898141264915466 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 ;"Purity/1FRt2SRqqt1FBt/event_purity‚0.014367816969752312 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.26661938428878784 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 ;"Purity/*FRt2SRqqt0FBt/event_purity‚0.020195212215185165 :"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.38983362913131714 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.4241935908794403 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 7 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.3377777338027954 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.2103174477815628 jet/accuracy_7_of_7‚0 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 9"Purity/2FRt0SRqqt*FBt/event_purity‚0.0420672707259655 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 + Purity/1FRt3SRqqt1FBt/FRt_purity‚0.4375 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.7142857313156128 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 ("Purity/0FRt2SRqqt3FBt/SRqqt_purity‚1 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.25486063957214355 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.7575758099555969 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 :"Purity/0FRt2SRqqt1FBt/event_purity‚0.03616352379322052 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 ;"Purity/0FRt1SRqqt2FBt/event_purity‚0.038382597267627716 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 7 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.3252873718738556 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.011044299229979515 6 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.281300812959671 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 epoch‚5 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9030733108520508 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.4725574851036072 7 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.2997903525829315 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 particle/accuracy_4_of_7‚1 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8518518209457397 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.26437652111053467 ,jet/accuracy_3_of_3‚0.031084464862942696 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 particle/accuracy_5_of_6‚1 *"Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0.5 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 :"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.41121411323547363 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.38689497113227844 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.3297262489795685 ;"Purity/*FRt3SRqqt*FBt/event_purity‚0.008064515888690948 :"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.35856711864471436 7 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.4333333373069763 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9875836968421936 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9915457367897034 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 9"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.3641975522041321 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 +particle/specificity‚0.9508225917816162 & Purity/0FRt2SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.3268490731716156 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.32705897092819214 7 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.3241041302680969 6 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.997527003288269 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 7 Purity/1FRt0SRqqt1FBt/FBt_purity‚0.9981725215911865 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.022012759000062943 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.22727273404598236 +jet/accuracy_4_of_6‚0.03846153989434242 7 Purity/3FRt4SRqqt0FBt/FRt_purity‚0.3333333432674408 particle/accuracy_5_of_7‚1 _step‚597 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5644201040267944 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 6 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.949999988079071 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5626728534698486 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.16525821387767792 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 :"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.32867860794067383 7 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.3141762614250183 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 *"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.5 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.3547143042087555 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9905178546905518 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 -"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.6875 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9765624403953552 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.6315789222717285 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.26254433393478394 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 8 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.35067567229270935 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 particle/accuracy_4_of_6‚1 jet/accuracy_1_of_7‚1 ( Purity/*FRt1SRqqt3FBt/FRt_purity‚0.5 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.16049189865589142 ;"Purity/*FRt3SRqqt0FBt/event_purity‚0.008064515888690948 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.3333333432674408 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 & Purity/1FRt3SRqqt2FBt/FRt_purity‚0 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 7 Purity/*FRt*SRqqt4FBt/FBt_purity‚0.8333333134651184 particle/accuracy_6_of_7‚1 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9823994040489197 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9981593489646912 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 6 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.190476194024086 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 9"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.4399999976158142 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 _runtime‚8476.731500692 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.30000001192092896 particle/accuracy_3_of_6‚1 & Purity/1FRt*SRqqt4FBt/FRt_purity‚0 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 8"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.596977949142456 7 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.3017241358757019 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.3723064363002777 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9818594455718994 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6482693552970886 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 particle/accuracy_5_of_5‚1 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 * Purity/4FRt*SRqqt*FBt/FRt_purity‚0.125 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9880322813987732 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.29943570494651794 ( Purity/2FRt*SRqqt2FBt/FRt_purity‚0.5 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 *jet/accuracy_2_of_4‚0.3815906047821045 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.2638888955116272 ;"Purity/0FRt3SRqqt*FBt/event_purity‚0.026143791154026985 ( Purity/1FRt1SRqqt3FBt/FRt_purity‚0.5 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9841046333312988 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9816563725471497 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.3829405605792999 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 :"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.42326393723487854 jet/accuracy_2_of_7‚0.5 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 trainer/global_step‚7253 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.3208836317062378 ;"Purity/2FRt*SRqqt2FBt/event_purity‚0.040697675198316574 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 +jet/accuracy_3_of_4‚0.06978162378072739 +jet/accuracy_4_of_5‚0.01941884122788906 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9920635223388672 :"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.47117355465888977 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 7 Purity/1FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5414940118789673 :"Purity/2FRt0SRqqt2FBt/event_purity‚0.04605263099074364 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_1_of_5‚1 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 & Purity/1FRt0SRqqt4FBt/FRt_purity‚0 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 7 Purity/2FRt0SRqqt1FBt/FBt_purity‚0.9940476417541504 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 7 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.4204545319080353 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 6validation_average_jet_accuracy‚0.4156716763973236 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.5127257704734802 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.3668108284473419 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 9"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.3741830289363861 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.30370453000068665 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 8"Purity/*FRt0SRqqt*FBt/event_purity‚0.426364004611969 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 +"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.25 :"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.40308496356010437 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 *jet/accuracy_3_of_5‚0.1274961531162262 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.17262554168701172 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.3055555522441864 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚1 :"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.36650556325912476 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.11519281566143036 9"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.3718573749065399 :"Purity/0FRt*SRqqt*FBt/event_purity‚0.41424956917762756 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.3278561234474182 particle/accuracy_3_of_4‚1 *jet/accuracy_2_of_6‚0.5064102411270142 7 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.2666666805744171 ;"Purity/*FRt1SRqqt2FBt/event_purity‚0.023022275418043137 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.25486063957214355 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.39459478855133057 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.06815475970506668 :"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.34098273515701294 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 *jet/accuracy_2_of_2‚0.1138594001531601 *jet/accuracy_1_of_3‚0.8632111549377441 particle/accuracy_2_of_3‚1 particle/accuracy_3_of_3‚1 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 :"Purity/*FRt*SRqqt1FBt/event_purity‚0.42278262972831726 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 +jet/accuracy_2_of_5‚0.48693040013313293 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.020244332030415535 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.30426543951034546 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 (particle/accuracy‚0.8557530641555786 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 5 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.80952388048172 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 :"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.36452236771583557 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.3541666567325592 % _timestamp‚1.7885188512588704e+09 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9984239935874939 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.34107109904289246 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 *"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.5 particle/accuracy_3_of_7‚1 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.20813468098640442 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5765029191970825 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9875239133834839 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 particle/accuracy_7_of_7‚1 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9892222285270691 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 :"Purity/0FRt*SRqqt0FBt/event_purity‚0.28393691778182983 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9993098378181458 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.39598217606544495 particle/accuracy_1_of_4‚1 +"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.25 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5543832778930664 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.29195886850357056 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.40833327174186707 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 *jet/accuracy_1_of_1‚0.4383634030818939 -"Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.3125 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 particle/accuracy_6_of_6‚1 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 ("Purity/4FRt*SRqqt0FBt/SRqqt_purity‚1 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 particle/accuracy_1_of_1‚1 particle/accuracy_1_of_2‚1 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 9"Purity/*FRt*SRqqt3FBt/event_purity‚0.1560283601284027 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.30763858556747437 + Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9375 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 6 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.878654956817627 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.4082627296447754 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9838408827781677 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.4057113230228424 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.4152844548225403 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 8"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.949999988079071 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9868749380111694 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.5257327556610107 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.3205931782722473 <"Purity/1FRt2SRqqt*FBt/event_purity‚0.0025012025143951178 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 :"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.43518519401550293 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 +"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.25 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.03004535287618637 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.18762743473052979 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.3995296061038971 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5465549826622009 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1º¬kà †§ :"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.40308496356010437 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 *jet/accuracy_3_of_5‚0.1274961531162262 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.17262554168701172 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.3055555522441864 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚1 :"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.36650556325912476 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.11519281566143036 9"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.3718573749065399 :"Purity/0FRt*SRqqt*FBt/event_purity‚0.41424956917762756 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.3278561234474182 particle/accuracy_3_of_4‚1 *jet/accuracy_2_of_6‚0.5064102411270142 7 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.2666666805744171 ;"Purity/*FRt1SRqqt2FBt/event_purity‚0.023022275418043137 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.25486063957214355 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.39459478855133057 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.06815475970506668 :"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.34098273515701294 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 *jet/accuracy_2_of_2‚0.1138594001531601 *jet/accuracy_1_of_3‚0.8632111549377441 particle/accuracy_2_of_3‚1 particle/accuracy_3_of_3‚1 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 :"Purity/*FRt*SRqqt1FBt/event_purity‚0.42278262972831726 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 +jet/accuracy_2_of_5‚0.48693040013313293 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.020244332030415535 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.30426543951034546 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 (particle/accuracy‚0.8557530641555786 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 5 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.80952388048172 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 :"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.36452236771583557 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.3541666567325592 % _timestamp‚1.7885188512588704e+09 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9984239935874939 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.34107109904289246 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 *"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.5 particle/accuracy_3_of_7‚1 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.20813468098640442 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5765029191970825 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9875239133834839 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 particle/accuracy_7_of_7‚1 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9892222285270691 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 :"Purity/0FRt*SRqqt0FBt/event_purity‚0.28393691778182983 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9993098378181458 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.39598217606544495 particle/accuracy_1_of_4‚1 +"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.25 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5543832778930664 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.29195886850357056 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.40833327174186707 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 *jet/accuracy_1_of_1‚0.4383634030818939 -"Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.3125 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 particle/accuracy_6_of_6‚1 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 ("Purity/4FRt*SRqqt0FBt/SRqqt_purity‚1 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 particle/accuracy_1_of_1‚1 particle/accuracy_1_of_2‚1 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 9"Purity/*FRt*SRqqt3FBt/event_purity‚0.1560283601284027 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.30763858556747437 + Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9375 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 6 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.878654956817627 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.4082627296447754 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9838408827781677 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.4057113230228424 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.4152844548225403 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 8"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.949999988079071 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9868749380111694 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.5257327556610107 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.3205931782722473 <"Purity/1FRt2SRqqt*FBt/event_purity‚0.0025012025143951178 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 :"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.43518519401550293 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 +"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.25 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.03004535287618637 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.18762743473052979 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.3995296061038971 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5465549826622009 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.4000000059604645 7 Purity/1FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9351851940155029 7 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.3039636015892029 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 +particle/sensitivity‚0.3419801592826843 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 * Purity/4FRt*SRqqt0FBt/FRt_purity‚0.125 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 ,jet/accuracy_4_of_4‚0.010287177748978138 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.2638888955116272 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 jet/accuracy_1_of_6‚1 particle/accuracy_2_of_6‚1 =&Purity/*FRt*SRqqt1FBt/event_wBwò"tproportion‚0.2714533805847168 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.6020832657814026 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.4750882387161255 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.30000001192092896 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 8"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.607730507850647 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.17977851629257202 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.5926619172096252 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 *jet/accuracy_2_of_3‚0.2542184293270111 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 + Purity/0FRt*SRqqt4FBt/FBt_purity‚0.8125 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 :"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.35344988107681274 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 jet/accuracy_6_of_6‚0 6 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.326577752828598 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 :"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.42467254400253296 :"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.37919387221336365 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.8571428656578064 )loss/val_total_loss‚39.21357400732002 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.38713526725769043 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 9"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.3297979533672333 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 :"Purity/*FRt2SRqqt1FBt/event_purity‚0.02241748757660389 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 ,"Purity/0FRt0SRqqt3FBt/event_purity‚0.125 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.9444444179534912 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.3541666567325592 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 ("Purity/*FRt2SRqqt3FBt/SRqqt_purity‚1 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 9"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.4000000059604645 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 8"Purity/0FRt*SRqqt2FBt/event_purity‚0.279652863740921 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 * Purity/1FRt0SRqqt3FBt/FRt_purity‚0.625 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.3238120973110199 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 8"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.618055522441864 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9977622032165527 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9974977374076843 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.3369171917438507 *jet/accuracy_1_of_4‚0.9094056487083435 & Purity/*FRt3SRqqt1FBt/FBt_purity‚1 +"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.25 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.2181648313999176 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.014971093274652958 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.2777777910232544 jet/accuracy_5_of_7‚0 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.11365262418985367 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.6388888955116272 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.5415721535682678 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 5 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.46123206615448 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.3010818362236023 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 particle/accuracy_1_of_3‚1 +jet/accuracy_3_of_6‚0.17307692766189575 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9700000286102295 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9893267154693604 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9901393055915833 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 )jet/accuracy_1_of_2‚0.733317494392395 & Purity/*FRt3SRqqt*FBt/FBt_purity‚1 jet/accuracy_4_of_7‚0 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.5417117476463318 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9952331185340881 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.32894736528396606 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.3363457918167114 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.3402777910232544 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9735663533210754 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.4253290295600891 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.7727272510528564 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9913323521614075 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.3333333134651184 particle/accuracy_1_of_6‚1 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.4642857015132904 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 ."Purity/1FRt0SRqqt3FBt/event_purity‚0.34375 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 6 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.301118940114975 particle/accuracy_2_of_4‚1 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 :"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.36772024631500244 8 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.32777780294418335 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.1291113793849945 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5534499883651733 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 7 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.3263888955116272 ("Purity/4FRt*SRqqt*FBt/SRqqt_purity‚1 :"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.37348467111587524 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9882693290710449 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.4174850881099701 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 particle/accuracy_1_of_7‚1 & Purity/*FRt*SRqqt4FBt/FRt_purity‚0 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 :"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.35643884539604187 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.31101956963539124 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9942528009414673 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 particle/accuracy_2_of_7‚1 & Purity/0FRt3SRqqt1FBt/FBt_purity‚1 7 Purity/0FRt2SRqqt1FBt/FBt_purity‚0.9811320900917053 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9079861044883728 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.3333333432674408 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.5000000596046448 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9832080006599426 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9896175265312195 7 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.2776707112789154 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 :"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.41604891419410706 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9766082167625427 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2688491940498352 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9179894328117371 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.17554017901420593 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.5000000596046448 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 jet/accuracy_6_of_7‚0 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.29579952359199524 <"Purity/1FRt1SRqqt2FBt/event_purity‚0.0069444444961845875 6 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.986111044883728 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 jet/accuracy_5_of_6‚0 & Purity/0FRt3SRqqt*FBt/FBt_purity‚1 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 ) Purity/4FRt1SRqqt0FBt/FRt_purity‚0.25 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 (particle/f_score‚0.42521730065345764 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.5115740895271301 +"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.25 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.4529609680175781 :"Purity/*FRt0SRqqt2FBt/event_purity‚0.28404977917671204 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.23822280764579773 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9993098378181458 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.3677172362804413 7 Purity/3FRt1SRqqt2FBt/FRt_purity‚0.3333333432674408 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.5106133818626404 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.3576986491680145 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.03900284320116043 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 7 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.3065843880176544 8"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.675438642501831 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.3333333432674408 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.3380413353443146 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.5515351295471191 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.3414141535758972 ,"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.875 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚1 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6000719666481018 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.07894736528396606 :"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.46666666865348816 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.2944486439228058 7 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.6176470518112183 5 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.32005575299263 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.32138946652412415 7 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.2953703999519348 7 Purity/3FRt4SRqqt*FBt/FRt_purity‚0.3333333432674408 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9951940178871155 + Purity/0FRt0SRqqt4FBt/FBt_purity‚0.8125 , Purity/*FRt2SRqqt2FBt/FRt_purity‚0.28125 jet/accuracy_3_of_7‚0 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 particle/accuracy_3_of_5‚1 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.04428185895085335 9"Purity/2FRt*SRqqt*FBt/event_purity‚0.0155824339017272 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 *jet/accuracy_1_of_5‚0.9473630785942078 :"Purity/0FRt1SRqqt0FBt/event_purity‚0.35344988107681274 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.3836187720298767 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 particle/accuracy_4_of_5‚1 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8500000238418579 +"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.25 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.05031375586986542 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 6 Purity/2FRt*SRqqt1FBt/FBt_purity‚0.998236358165741 particle/accuracy_2_of_5‚1 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.4340277910232544 :"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.40740740299224854 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.7035030126571655 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 ;"Purity/0FRt2SRqqt0FBt/event_purity‚0.046031802892684937 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.1187903881072998 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 8"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.459770143032074 ;"Purity/0FRt3SRqqt0FBt/event_purity‚0.026143791154026985 7 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.2993900179862976 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8197475075721741 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.3214285671710968 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 ) Purity/4FRt1SRqqt*FBt/FRt_purity‚0.25 8 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.48517507314682007 8 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.42111989855766296 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 jet/accuracy_5_of_5‚0 particle/accuracy_4_of_4‚1 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.3645833432674408 *"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.5 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.5430143475532532 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 particle/accuracy_2_of_2‚1 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 & Purity/*FRt0SRqqt4FBt/FRt_purity‚0 & Purity/*FRt3SRqqt2FBt/FRt_purity‚0 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.5922618508338928 :"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.40898141264915466 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 ;"Purity/1FRt2SRqqt1FBt/event_purity‚0.014367816969752312 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.26661938428878784 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 ;"Purity/*FRt2SRqqt0FBt/event_purity‚0.020195212215185165 :"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.38983362913131714 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.4241935908794403 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 7 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.3377777338027954 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.2103174477815628 jet/accuracy_7_of_7‚0 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 9"Purity/2FRt0SRqqt*FBt/event_purity‚0.0420672707259655 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 + Purity/1FRt3SRqqt1FBt/FRt_purity‚0.4375 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.7142857313156128 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 ("Purity/0FRt2SRqqt3FBt/SRqqt_purity‚1 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.25486063957214355 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.7575758099555969 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 :"Purity/0FRt2SRqqt1FBt/event_purity‚0.03616352379322052 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 ;"Purity/0FRt1SRqqt2FBt/event_purity‚0.038382597267627716 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 7 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.3252873718738556 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.011044299229979515 6 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.281300812959671 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 epoch‚5 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9030733108520508 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.4725574851036072 7 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.2997903525829315 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 particle/accuracy_4_of_7‚1 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8518518209457397 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.26437652111053467 ,jet/accuracy_3_of_3‚0.031084464862942696 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 particle/accuracy_5_of_6‚1 *"Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0.5 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 :"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.41121411323547363 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.38689497113227844 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.3297262489795685 ;"Purity/*FRt3SRqqt*FBt/event_purity‚0.008064515888690948 :"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.35856711864471436 7 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.4333333373069763 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9875836968421936 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9915457367897034 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 9"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.3641975522041321 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 +particle/specificity‚0.9508225917816162 & Purity/0FRt2SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.3268490731716156 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.32705897092819214 7 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.3241041302680969 6 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.997527003288269 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 7 Purity/1FRt0SRqqt1FBt/FBt_purity‚0.9981725215911865 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.022012759000062943 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.22727273404598236 +jet/accuracy_4_of_6‚0.03846153989434242 7 Purity/3FRt4SRqqt0FBt/FRt_purity‚0.3333333432674408 particle/accuracy_5_of_7‚1 _step‚597 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5644201040267944 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 6 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.949999988079071 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5626728534698486 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.16525821387767792 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 :"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.32867860794067383 7 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.3141762614250183 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 *"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.5 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.3547143042087555 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9905178546905518 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 -"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.6875 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9765624403953552 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.6315789222717285 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.26254433393478394 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 8 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.35067567229270935 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 particle/accuracy_4_of_6‚1 jet/accuracy_1_of_7‚1 ( Purity/*FRt1SRqqt3FBt/FRt_purity‚0.5 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.16049189865589142 ;"Purity/*FRt3SRqqt0FBt/event_purity‚0.008064515888690948 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.3333333432674408 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 & Purity/1FRt3SRqqt2FBt/FRt_purity‚0 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 7 Purity/*FRt*SRqqt4FBt/FBt_purity‚0.8333333134651184 particle/accuracy_6_of_7‚1 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9823994040489197 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9981593489646912 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 6 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.190476194024086 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 9"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.4399999976158142 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 _runtime‚8476.731500692 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.30000001192092896 particle/accuracy_3_of_6‚1 & Purity/1FRt*SRqqt4FBt/FRt_purity‚0 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 8"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.596977949142456 7 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.3017241358757019 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.3723064363002777 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9818594455718994 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6482693552970886 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 particle/accuracy_5_of_5‚1 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 * Purity/4FRt*SRqqt*FBt/FRt_purity‚0.125 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9880322813987732 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.29943570494651794 ( Purity/2FRt*SRqqt2FBt/FRt_purity‚0.5 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 *jet/accuracy_2_of_4‚0.3815906047821045 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.2638888955116272 ;"Purity/0FRt3SRqqt*FBt/event_purity‚0.026143791154026985 ( Purity/1FRt1SRqqt3FBt/FRt_purity‚0.5 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9841046333312988 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9816563725471497 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.3829405605792999 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 :"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.42326393723487854 jet/accuracy_2_of_7‚0.5 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 trainer/global_step‚7253 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.3208836317062378 ;"Purity/2FRt*SRqqt2FBt/event_purity‚0.040697675198316574 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 +jet/accuracy_3_of_4‚0.06978162378072739 +jet/accuracy_4_of_5‚0.01941884122788906 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9920635223388672 :"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.47117355465888977 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 7 Purity/1FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5414940118789673 :"Purity/2FRt0SRqqt2FBt/event_purity‚0.04605263099074364 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_1_of_5‚1 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 & Purity/1FRt0SRqqt4FBt/FRt_purity‚0 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 7 Purity/2FRt0SRqqt1FBt/FBt_purity‚0.9940476417541504 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 7 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.4204545319080353 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 6validation_average_jet_accuracy‚0.4156716763973236 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.5127257704734802 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.3668108284473419 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 9"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.3741830289363861 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.30370453000068665 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 8"Purity/*FRt0SRqqt*FBt/event_purity‚0.426364004611969 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 +"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.25ÕûÓhç”á ‚(:‰ ÐÃêÔŽËêÿ gpu.0.gpu‚43.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚157.243' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚39.31075 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0°ƒSmâ ‚(:â ÐÃêÔšŠÒ€ disk.sda2.out‚972.5498046875 disk.md0.in‚ 9.26953125 disk.sda1.in‚0( proc.memory.availableMB‚ 1295.5234375 disk.sda.in‚ 6.08203125 disk.rbd0.out‚ 2.466796875 disk.md0.out‚ 937.31640625 disk.dm-0.out‚94648.00390625$ memory_percent‚92.09275245666504) proc.memory.percent‚18.31347942352295 disk.sda1.out‚0 network.recv‚ 2.747324e+06 disk.md0p1.out‚ 937.31640625 disk.nvme0n1.in‚ 603.58984375 disk.sdb.in‚3.1875 disk.sdb1.out‚0 disk.dm-0.in‚ 1102.390625# proc.memory.rssMB‚ 3000.48046875% disk./.usageGB‚2713.1197242736816 proc.cpu.threads‚17 disk.sda.out‚972.5498046875 disk.nvme1n1.in‚ 498.80078125# disk.nvme1n1.out‚47672.36328125 disk.rbd2.in‚0 disk.rbd2.out‚ 2.466796875! disk.nvme0n1.out‚ 46975.640625 disk.sda2.in‚ 6.08203125 disk.rbd0.in‚0 disk.sdb2.in‚3.1875 disk.sdb2.out‚972.5498046875 network.sent‚ 1.8561704e+07 disk.sdb.out‚972.5498046875 disk.sdb1.in‚0 cpu‚25.217523215545018) disk./.usagePercent‚75.85996301951359 disk.md0p1.in‚ 9.26953125ÞÏ+”ã ‚(:‰ ßÃêÔª»Ÿÿ gpu.0.gpu‚47.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚37.0 gpu.0.powerWatts‚171.777' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.94425 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ûg°û&ä ‚(:ß ßÃêÔ¿„¦€ disk.nvme0n1.in‚ 603.58984375! disk.sda2.out‚974.48095703125$ memory_percent‚92.09227561950684# proc.memory.rssMB‚ 3000.48046875) disk./.usagePercent‚75.86695312414672 disk.md0.out‚939.1875 disk.md0p1.in‚ 9.26953125 disk.sda2.in‚ 6.08203125 proc.cpu.threads‚17 network.sent‚ 1.8687344e+07 disk.md0.in‚ 9.26953125 disk.sda1.in‚0" disk.nvme0n1.out‚ 47127.8046875 disk.sda.out‚974.48095703125) proc.memory.percent‚18.31347942352295 disk.md0p1.out‚939.1875( proc.memory.availableMB‚ 1295.6015625 disk.nvme1n1.in‚ 498.80078125# disk.nvme1n1.out‚47672.65234375 disk.sdb1.in‚0 cpu‚25.217172076818084 network.recv‚ 2.752776e+06 disk.sda1.out‚0 disk.dm-0.in‚ 1102.390625 disk.sdb.in‚3.1875 disk.sdb.out‚974.48095703125 disk.sdb2.in‚3.1875! disk.sdb2.out‚974.48095703125 disk.sda.in‚ 6.08203125 disk.rbd0.in‚0 disk.rbd2.in‚0 disk.rbd2.out‚ 2.466796875% disk./.usageGB‚2713.3697242736816 disk.rbd0.out‚JÎøD 2.466796875 disk.sdb1.out‚0 disk.dm-0.out‚94800.45703125qõžÍiå ‚(:Þ îÃêÔ°ÃÓÿ# proc.memory.rssMB‚ 3000.48046875 proc.cpu.threads‚17 network.recv‚ 2.757671e+06 disk.sdb.in‚3.1875 disk.md0.in‚ 9.26953125 disk.dm-0.out‚94949.69921875! disk.nvme1n1.out‚ 47672.796875 disk.sda1.out‚0 disk.md0p1.out‚ 941.015625 disk.nvme0n1.in‚ 603.58984375% disk./.usageGB‚2713.3697242736816 disk.sda2.out‚976.3837890625 disk.rbd0.in‚0 disk.sdb1.out‚0 disk.dm-0.in‚ 1102.390625 disk.sdb1.in‚0 disk.md0.out‚ 941.015625 disk.rbd2.in‚0 disk.rbd2.out‚ 2.466796875 disk.sda.in‚ 6.08203125 cpu‚25.216794920919952) disk./.usagePercent‚75.86695312414672 disk.rbd0.out‚ 2.466796875 disk.nvme1n1.in‚ 498.80078125 disk.sdb2.out‚976.3837890625 disk.sda1.in‚0) proc.memory.percent‚18.31347942352295 network.sent‚ 1.869362e+07 disk.sdb2.in‚3.1875$ memory_percent‚92.09449291229248 disk.sda2.in‚ 6.08203125 disk.md0p1.in‚ 9.26953125# disk.nvme0n1.out‚47276.90234375 disk.sdb.out‚976.3837890625 disk.sda.out‚976.3837890625) proc.memory.availableMB‚ 1295.23828125s®(“æ ‚(:ˆ îÃêÔÓ“ùÿ gpu.0.gpu‚41.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚169.942' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.4855 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0JëG2ç - _wandbruntime‚8527  _runtime‚8527ƒšñ™Éè à % _timestamp‚1.7885189022414334e+09 _runtime‚8527.711858853 _step‚598 ' lr-AdamW/pg1‚0.00034890046723042847 ' lr-AdamW/pg2‚0.00034890046723042847 trainer/global_step‚7299t\ð<Îé È _runtime‚8527.711858853 _step‚598 ' lr-AdamW/pg1‚0.00034890046723042847 ' lr-AdamW/pg2‚0.00034890046723042847 trainer/global_step‚7299 % _timestamp‚1.7885189022414334e+09ÖÈÁ92ê - _wandbruntime‚8527  _runtime‚8527)ü¢ëü'ë öO YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚596369270081205 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚588089341173392 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 151040382 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚609886492786176 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407872 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚599 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚601576503613952 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚43938862 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 151040384 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177047040 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1212 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 trainer/global_step‚7299 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 334361976 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚609886492786176 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8309989172224 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚93919173 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917410816 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚624 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8676044122624 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8675817359872 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450780 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 183321594 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚609886492786176 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407872 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚599 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚601576503613952 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚43938862 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 151040384 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177047040 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1212 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 trainer/global_step‚7299 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 334361976 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚609886492786176 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8309989172224 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚93919173 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171׃³‹Ï2í - _wandbruntime‚8528  _runtime‚8528v‡¯Ë'î €N ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 137876152 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007241 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 334406465 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899421184 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚673099854215680 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 trainer/global_step‚7299 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 151060282 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚609968153309184 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1187 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8281020529979 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 183346185 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 137876782 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚596447444385871 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _step‚600 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚601656151912448 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177047040 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚588167338306880 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 334432581 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚601656133909504 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 334432581 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚628 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚596448358836859 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8311101975552 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8311101975552 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31686 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚601656151912448 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚588167321267520 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917424128 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚43945231 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚43944603 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚681777057205248 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚601656133909504 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177047040 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 233040896 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 183372299 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚93931549 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0¢›¬<ï …N C=DeviceStatsMonitor.on_train_batch_end/oversize_seg(‰HÏ&ments.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411628 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8312001396736 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 137876152 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007241 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 334406465 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899421184 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚673099854215680 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 trainer/global_step‚7299 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 151060282 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚609968153309184 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1187 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8281020529979 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 183346185 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 137876782 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚596447444385871 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _step‚600 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚601656151912448 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177047040 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚588167338306880 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 334432581 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚601656133909504 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 334432581 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚628 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚596448358836859 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8311101975552 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8311101975552 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31686 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚601656151912448 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚588167321267520 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917424128 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚43945231 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚43944603 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚681777057205248 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚601656133909504 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177047040 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 233040896 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 183372299 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚93931549 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 EDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214665 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 184554496 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚94537975 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 184580584 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917410816 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚600 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955591168 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚600322806993933 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 _runtime‚8588.578188166 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚613923662144000 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 152035282 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚591988362919152 % _timestamp‚1.7885189631076822e+09 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450780 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚605558018391552 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚613923662144000 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8733984968704<|»—è €!ûO N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 138759167 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚94537973 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407872 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 184580584 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1187 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8334461114141 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411420 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 152035284 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚605558036394496 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26088 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8366543160320 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚677476496031744 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚44221194 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 336615868 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007241 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8365643752448 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚605558036394496 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8733760303104 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8335358525561 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚613923662144000 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 152035282 `LDeviceStatsMonitor.on_qRgtrain_batch_start/requested_bytes.large_pool.allocated‚591988362919152 % _timestamp‚1.7885189631076822e+09 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450780 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚605558018391552 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚613923662144000 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8733984968704 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚613924579554816 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 138759767 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚605558018391552 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚677476472091648 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 224665600 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 152035282 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 399863808 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214665 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 184554496 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚94537975 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 184580584 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917410816 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚600 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955591168 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚600322806993933 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 _runtime‚8588.578188166 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 184605201 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚600402810200367 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 152055182 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177047040 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚600401895749379 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955591168 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899421184 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177047040 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 _runtime‚8589.525895343 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚605637648687104 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 336634267 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 184579087 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 399863808 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8335569775459 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 152055180 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚605637666690048 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899421184 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 152055182 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411628 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚605637666690048 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 184605201 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 trainer/global_step‚7349 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 152055180 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917424128 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 184579087 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 138777205 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8735148924928 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚94550351 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚44226856 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚44227481 % _timestamp‚1.7885189640549748e+09 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚614005322667008 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26114 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31686 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚677565815252480 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚605637648687104 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚686300964177408 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚592066343013280 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 250689536 _step‚604 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 336660383 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 226749440 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450988 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 336634267`£Î  'ƒ!…N REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899421184 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 152055182 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411628 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚605637666690048 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 184605201 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 trainer/global_step‚7349 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 152055180 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917424128 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 184579087 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 138777205 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8735148924928 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚94550351 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚44226856 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚44227481 % _timestamp‚1.7885189640549748e+09 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚614005322667008 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26114 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31686 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚677565815252480 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚605637648687104 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚686300964177408 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚592066343013280 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 250689536 _step‚604 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 336660383 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 226749440 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450988 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 336634267 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 184605201 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚600402810200367 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 152055182 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177047040 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚600401895749379 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955591168 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899421184 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177047040 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 _runtime‚8589.525895343 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚605637648687104 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 336634267 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 184579087 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 399863808 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8335569775459 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 152055180 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚605637666690048 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008Ü3^2„!- _wandbruntime‚8589  _runtime‚8589lÔZ…!Ô $loss/total_loss‚40.7551993485143 0loss/FBt1/assignment_loss‚0.4596298472948547 1loss/SRqqt4/detection_loss‚2.3680435474961996 2loss/SRqqt3/assignment_loss‚1.2338830282830673 /loss/FRt3/detection_loss‚2.2268241557810042 /loss/FBt2/detection_loss‚0.6396169111831114 2loss/SRqqt1/assignment_loss‚1.1902778226022537 0loss/FBt3/assignment_loss‚0.7619519705172024 1loss/SRqqt3/detection_loss‚1.4947621843413166 _runtime‚8589.527451399 .loss/FRt4/detection_loss‚2.905562363821885 /loss/FBt1/detection_loss‚2.5068583607673647 $ _timestamp‚1.788518964057322e+09 0loss/FRt1/assignment_loss‚2.9002618833286005 0loss/SRqqt1/detection_loss‚1.969237594650342 /loss/FBt3/detection_loss‚3.3712973395983377 epoch‚6 /loss/FRt1/detection_loss‚2.3213229047353416 0loss/FRt2/assignment_loss‚1.2652784819696463 .loss/FRt3/assignment_loss‚2.02886770301395 0loss/FBt4/assignment_loss‚0.2719241255981615 1loss/SRqqt2/detection_loss‚1.1404131037803988 trainer/global_step‚7349 _step‚605 0loss/FRt4/assignment_loss‚2.5007941106270097 2loss/SRqqt2/assignment_loss‚0.8857519121633636 2loss/SRqqt4/assignment_loss‚1.7058797816280276 1loss/FBt2/assignment_loss‚0.05050800071352207 /loss/FRt2/detection_loss‚1.7285107705084717 /loss/FBt4/detection_loss‚2.8277414441108704 cUí_†!Ù 0loss/FRt1/assignment_loss‚2.9002618833286005 0loss/SRqqt1/detection_loss‚1.969237594650342 /loss/FBt3/detection_loss‚3.3712973395983377 epoch‚6 /loss/FRt1/detection_loss‚2.3213229047353416 0loss/FRt2/assignment_loss‚1.2652784819696463 .loss/FRt3/assignment_loss‚2.02886770301395 0loss/FBt4/assignment_loss‚0.2719241255981615 1loss/SRqqt2/detection_loss‚1.1404131037803988 trainer/global_step‚7349 _step‚605 0loss/FRt4/assignment_loss‚2.5007941106270097 2loss/SRqqt2/assignment_loss‚0.8857519121633636 2loss/SRqqt4/assignment_loss‚1.7058797816280276 1loss/FBt2/assignment_loss‚0.05050800071352207 /loss/FRt2/detection_loss‚1.7285107705084717 /loss/FBt4/detection_loss‚2.8277414441108704 $loss/total_loss‚40.7551993485143 0loss/FBt1/assignment_loss‚0.4596298472948547 1loss/SRqqt4/detection_loss‚2.3680435474961996 2loss/SRqqt3/assignment_loss‚1.2338830282830673 /loss/FRt3/detection_loss‚2.2268241557810042 /loss/FBt2/detection_loss‚0.6396169111831114 2loss/SRqqt1/assignment_loss‚1.1902778226022537 0loss/FBt3/assignment_loss‚0.7619519705172024 1loss/SRqqt3/detection_loss‚1.4947621843413166 _runtime‚8589.527451399 .loss/FRt4/detection_loss‚2.905562363821885 /loss/FBt1/detection_loss‚2.5068583607673647 $ _timestamp‚1.788518964057322e+09Ý^•›}”‡!‚(:‰ ¹ÄêÔ÷¸Â€ gpu.0.gpu‚11.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚36.0 gpu.0.powerWatts‚141.087' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚35.27175 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0=Ó2†gˆ!‚(:Ü ¹ÄêÔ‰§† proc.cpu.threads‚17 disk.dm-0.in‚ 1102.390625 disk.dm-0.out‚95826.65234375# disk.nvme0n1.out‚48182.95703125 disk.sdb.in‚3.1875 disk.sda.out‚985.6884765625 disk.nvme0n1.in‚ 603.58984375 disk.sda2.out‚985.6884765625# proc.memory.rssMB‚ 3000.48046875 network.recv‚ 2.77853e+06) disk./.usagePercent‚75.88093717319214 disk.sda1.out‚0 disk.sda.in‚ 6.08203125 disk.rbd0.in‚0" disk.nvme1n1.out‚ 47674.1953125 disk.sdb1.in‚0 disk.md0.out‚ 949.98046875 disk.sda2.in‚ 6.08203125# proc.memory.availableMB‚1294.75 cpu‚25.215132158842135 disk.sdb2.out‚985.6884765625 disk.sdb.out‚985.6884765625 disk.rbd0.out‚ 2.466796875$ memory_percent‚92.09747314453125 network.sent‚ 1.8821927e+07$ disk./.usageGB‚2713.869861602783 disk.sdb1.out‚0 disk.md0.in‚ 9.26953125 disk.sdb2.in‚3.1875 disk.sda1.in‚0 disk.md0p1.in‚ 9.26953125 disk.nvme1n1.in‚ 498.80078125 disk.rbd2.in‚0 disk.rbd2.out‚ 2.466796875 disk.md0p1.out‚ 949.98046875) proc.memory.percent‚18.31347942352295–´z8”‰!‚(:‰ ÈÄêÔîâäþ gpu.0.gpu‚54.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚164.039' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚41.00975 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Tx0ÝŠ!‚(:à ÈÄêÔװ׃ disk.sda.out‚987.13134765625! disk.sda2.out‚987.13134765625 network.sent‚ 1.8930881e+07( disk./.usagePercent‚75.8809350399815# disk.nvme0n1.out‚48301.76953125 disk.sdb.in‚3.1875 disk.sdb1.out‚0) proc.memory.availableMB‚ 1295.00390625 disk.rbd0.in‚0 disk.nvme1n1.in‚ 498.80078125 disk.md0.in‚ 9.26953125 disk.sdb2.in‚3.1875! disk.sdb2.out‚987.13134765625$ memory_percent‚92.09592342376709 cpu‚25.21477170695411 proc.cpu.threads‚17$ disk./.usageGB‚2713.869785308838 disk.nvme0n1.in‚ 603.58984375 disk.rbd0.out‚ 2.466796875# disk.nvme1n1.out‚47674.41796875 disk.rbd2.out‚ 2.466796875 disk.sda1.in‚0# proc.memory.ròpï+ŽssMB‚ 3000.48046875 disk.md0p1.out‚ 951.3515625 disk.sdb.out‚987.13134765625 disk.sda2.in‚ 6.08203125 disk.dm-0.in‚ 1102.390625 disk.md0.out‚ 951.3515625 network.recv‚ 2.783391e+06) proc.memory.percent‚18.31347942352295 disk.md0p1.in‚ 9.26953125 disk.rbd2.in‚0 disk.sda1.out‚0 disk.sda.in‚ 6.08203125 disk.dm-0.out‚ 95976.1875 disk.sdb1.in‚0øãÌ,’‹!‚(:‡ ×ÄêÔìðÿ gpu.0.gpu‚49.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚37.0 gpu.0.powerWatts‚163.09' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚40.7725 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0f$lhqŒ!‚(:æ ×ÄêÔ²µ±€! disk.sdb2.out‚989.14697265625 disk.sda.out‚989.14697265625 disk.dm-0.out‚96204.45703125! disk.nvme0n1.out‚ 48529.921875 disk.rbd2.in‚0 network.recv‚ 2.787098e+06( disk./.usagePercent‚75.8514751876884 disk.md0.out‚ 953.31640625 disk.sda1.out‚0 disk.nvme0n1.in‚ 603.58984375 disk.sda2.in‚ 6.08203125! disk.sda2.out‚989.14697265625 disk.rbd0.in‚0$ memory_percent‚92.09668636322021) proc.memory.percent‚18.31347942352295 disk.nvme1n1.in‚ 498.80078125 disk.md0.in‚ 9.26953125 disk.sda.in‚ 6.08203125# proc.memory.rssMB‚ 3000.48046875% disk./.usageGB‚2712.8161582946777 disk.rbd2.out‚ 2.466796875 disk.sdb2.in‚3.1875 disk.sdb.out‚989.14697265625 cpu‚25.214476808868397 proc.cpu.threads‚17 disk.md0p1.out‚ 953.31640625 disk.sdb.in‚3.1875 network.sent‚ 1.8936791e+07 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.md0p1.in‚ 9.26953125 disk.rbd0.out‚ 2.466796875# disk.nvme1n1.out‚47674.53515625 disk.sda1.in‚0 disk.dm-0.in‚ 1102.390625) proc.memory.availableMB‚ 1294.87890625GLè‘!‚(:† æÄêÔÇÁ¦ÿ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚175.512' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.878 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0™«‚û[Ž!‚(:Ð æÄêÔä¡‹€ disk.rbd2.in‚0 network.sent‚ 1.8943177e+07 disk.sda1.in‚0 disk.sda1.out‚0" disk.nvme0n1.out‚ 48601.8046875 disk.rbd0.out‚ 2.466796875! disk.nvme1n1.out‚ 47674.734375 disk.md0.out‚954.9375 disk.sdb1.in‚0 disk.md0.in‚ 9.26953125 disk.sdb.in‚3.1875 disk.sdb.out‚ 990.83203125 disk.sda.out‚ 990.83203125 disk.sdb2.in‚3.1875 cpu‚25.214136225489444 disk.md0p1.in‚ 9.26953125) proc.memory.percent‚18.31347942352295% disk./.usageGB‚2712.8161010742188 disk.sda2.in‚ 6.08203125$ memory_percent‚92.09532737731934# proc.memory.rssMB‚ 3000.48046875 network.recv‚ 2.791747e+06 proc.cpu.threads‚17) disk./.usagePercent‚75.85147358778043 disk.dm-0.in‚ 1102.390625 disk.dm-0.out‚ 96276.5390625 disk.md0p1.out‚954.9375 disk.rbd0.in‚0 disk.sda.in‚ 6.08203125 disk.nvme1n1.in‚ 498.80078125 disk.sdb1.out‚0 disk.sdb2.out‚ 990.83203125 disk.nvme0n1.in‚ 603.58984375 disk.sda2.out‚ 990.83203125 disk.rbd2.out‚ 2.466796875( proc.memory.availableMB‚ 1295.10156251`ì2!- _wandbruntime‚8643  _runtime‚8643GNb¯Ç!Á & lr-AdamW/pg1‚0.0003367706651429634 & lr-AdamW/pg2‚0.0003367706651429634 trainer/global_step‚7399 % _timestamp‚1.7885190183232982e+09 _runtime‚8643.793775432 _step‚6067hâÌ‘!Æ % _timestamp‚1.7885190183232982e+09 _runtime‚8643.793775432 _step‚606 & lr-AdamW/pg1‚0.0003367706651429634 & lr-AdamW/pg2‚0.0003367706651429634 trainer/global_step‚7399Þâ“ 2’!- _wandbruntime‚8643  _runtime‚8643dBÍöü'“!öO H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8421298332672 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 338843670 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 _step‚607 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407872 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚613 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚604277258357441 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26088 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚44507421 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 252800000 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚595887367625552 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚617961748912640 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 185787398 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214665 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 185813486 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 185787398 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 153030184 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚95156773 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚604278172808221 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26090 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚44506810 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚617961748912640 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8421298332672 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚609539533169152 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚617960831501824 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410816 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 _runtime‚8643.794581086 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 153030184 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 185813486 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚609539533169152 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407872 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 139663583 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚595887350586192 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955591168 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 228859904 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26088 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8422197740544 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450780 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 % _timestamp‚1.7885190183240838e+09 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚617960831501824 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410816 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 _runtime‚8643.794581086 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 153030184 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 185813486 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚609539533169152 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407872 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 139663583 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚595887350586192 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955591168 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 228859904 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26088 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8422197740544 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450780 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 % _timestamp‚1.7885190183240838e+09 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8421298332672 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 338843670 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 _step‚607 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407872 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚613 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚604277258357441 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26088 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚44507421 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 252800000 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚595887367625552 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚617961748912640 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 185787398 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214665 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 185813486 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 185787398 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 153030184 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚95156773 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚604278172808221 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26090 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚44506810 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚617961748912640 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8421298332672 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚609539533169152 JDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 338888185 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 139681623 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 185811989 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚618041574600704 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8391016432567 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 trainer/global_step‚7399 _runtime‚8644.756955527 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚44513092 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚44512474 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 399863808 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚604357261563875 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚609619181467648 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899421184 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26114 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚682031776289280 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411628 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 185838103 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 226749440 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8792842515456 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚609619163464704 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8422411136000 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 153050082 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 139682243 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 153050080 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31686 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚604356347112887 _step‚608 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8422411136000 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚682031752349184 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8423310557184 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450988 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917424128 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899421184 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚620 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚690824845554176 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚595965330680320 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 $ _timestamp‚1.788519019285659e+09 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚618 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007241 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚595965347719680&v• '—!„N TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 250689536 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚95169149 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 185838103 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 139681623 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 185811989 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚618041574600704 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8391016432567 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 trainer/global_step‚7399 _runtime‚8644.756955527 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚44513092 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚44512474 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 399863808 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚604357261563875 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚609619181467648 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899421184 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26114 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚682031776289280 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411628 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 185838103 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 226749440 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8792842515456 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚609619163464704 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8422411136000 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 153050082 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 139682243 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 153050080 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31686 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚604356347112887 _step‚608 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8422411136000 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚682031752349184 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8423310557184 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450988 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917424128 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899421184 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚620 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚690824845554176 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚595965330680320 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 $ _timestamp‚1.788519019285659e+09 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚618 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007241 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚595965347719680 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚95169151 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 338862069 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 338888185 Bocü'¥!öO trainer/global_step‚7449 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚608232624171729 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8446251839777 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚621998000859648 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚95775573 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8477852320768 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8849643473408 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955591168 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚695258037638656 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚599786372331952 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 _runtime‚8696.585285465 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 341071472 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚686408418105344 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8445354428357 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8477852320768 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 140568009 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 140567397 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚44792434 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 187046388 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1187 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 154025082 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚612 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26088 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 341045382 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 187020300 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8476952912896 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _step‚611 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26090 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚613521047946752 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450780 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917410816 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚44791824 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8849876527616 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 187046388 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 187020300 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214665 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚621998918270464 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚621998000859648 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚613521065949696 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 256994304 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26090 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚613521047946752 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450780 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917410816 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚44791824 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8849876527616 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 187046388 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 187020300 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214665 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚621998918270464 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚621998000859648 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚613521065949696 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 256994304 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 341115987 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 187044891 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚613600678242304 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31686 % _timestamp‚1.7885190720771837e+09 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 341115987 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 _runtime‚8697.548324665 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚613600696245248 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26114 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚95787951 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚95787949 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚44798013 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8478065716224 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 140585354 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 341089871 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 341089871 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚613600678242304 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚686497713385984 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 187044891 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _step‚612 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8478965137408 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899421184 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 341115987 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 187044891 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚613600678242304 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31686 % _timestamp‚1.7885190720771837e+09 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 341115987 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 _runtime‚8697.548324665 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚613600696245248 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26114 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚95787951 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚95787949 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚44798013 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8478065716224 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 140585354 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 341089871 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 341089871 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚613600678242304 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚686497713385984 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 187044891 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _step‚612 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8478965137408 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899421184 FDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚626035170217472 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8907919337984 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955591168 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚617502562724352 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 343299274 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 155019984 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411420 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚607 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚699782298480128 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚690874379142144 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 343273184 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚617502580727296 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450780 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚626036087628288 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407872 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚605 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 155019984 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 188253202 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407872 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 155019982 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚96394375 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8501698496885 % _timestamp‚1.7885191239890897e+09 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚626035170217472 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8532607493120 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26088 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 _step‚615 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8907692575232 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚612186161084457 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917410816 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 188253202 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚690874355202048Îpbî(º!ûO WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚607 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚699782298480128 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚690874379142144 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 343273184 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚617502580727296 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450780 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚626036087628288 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407872 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚605 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 155019984 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 188253202 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407872 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 155019982 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚96394375 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8501698496885 % _timestamp‚1.7885191239890897e+09 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚626035170217472 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8532607493120 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26088 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 _step‚615 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8907692575232 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚612186161084457 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917410816 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 188253202 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚690874355202048 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚699782047777280 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214665 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 188279290 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177047040 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚45077610 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚612187075535237 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚617502580727296 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚626035170217472 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8907919337984 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955591168 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚617502562724352 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 343299274 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 155019984 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411420ç E©h2»!- _wandbruntime‚8750  _runtime‚8750.îzè¾¼!ÿM S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚626115913316352 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1187 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚699872521553920 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚612266164290891 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 343343789 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 155039880 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 399863808 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚617582193019904 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899421184 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚633 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8534619717632 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 141490064 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8501909746783 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 188277793 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 155039882 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 _step‚616 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899421184 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8533720296448 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚45083946 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚699872776437760 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚617582211022848 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8909078074880 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8502807158411 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.curr¸“ÖLGent‚ 254883840 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚690963698362880 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007241 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 trainer/global_step‚7499 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚690963674422784 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 155039880 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚603763340093120 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 _runtime‚8750.422395268 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚617582193019904 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 343317673 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚603763357132480 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚617582211022848 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450988 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚612265249839903$€8x '½!„N VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 155039882 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 _step‚616 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899421184 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8533720296448 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚45083946 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚699872776437760 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚617582211022848 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8909078074880 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8502807158411 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 254883840 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚690963698362880 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007241 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 trainer/global_step‚7499 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚690963674422784 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 155039880 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚603763340093120 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 _runtime‚8750.422395268 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚617582193019904 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 343317673 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚603763357132480 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚617582211022848 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450988 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚612265249839903 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚626115913316352 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1187 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚699872521553920 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚612266164290891 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 343343789 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 155039880 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 399863808 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚617582193019904 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899421184 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚633 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8534619717632 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 141490064 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8501909746783 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 188277793 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607èÜ·2¾!- _wandbruntime‚8750  _runtime‚8750i¯fZ¿!Ô %loss/total_loss‚41.09645210566584 1loss/SRqqt4/detection_loss‚2.4620543365125305 /loss/FBt2/detection_loss‚0.5922910557443125 _step‚617 0loss/FRt4/assignment_loss‚2.4115000983079273 /loss/FBt4/detection_loss‚2.3565572930706873 /loss/FRt1/assignment_loss‚2.883172892942661 0loss/FRt3/assignment_loss‚2.8939358845670173 /loss/FRt2/detection_loss‚1.9232337770935817 epoch‚6 0loss/FRt2/assignment_loss‚1.7144326475950389 1loss/SRqqt3/assignment_loss‚1.229278185233778 0loss/SRqqt2/detection_loss‚1.188205051220587 1loss/FBt3/assignment_loss‚0.40830380421248264 .loss/FRt1/detection_loss‚2.261957817629392 /loss/FBt3/detection_loss‚3.4668107986450196 trainer/global_step‚7499 2loss/SRqqt2/assignment_loss‚0.9669220603269632 2loss/SRqqt4/assignment_loss‚1.6261439163375784 0loss/FBt4/assignment_loss‚0.0781949981335654 $ _timestamp‚1.788519124953305e+09 2loss/SRqqt1/assignment_loss‚1.2808993353694678 1loss/FBt1/assignment_loss‚0.23250522496865805 1loss/FBt2/assignment_loss‚0.10025641126070836 .loss/FRt3/detection_loss‚2.295122797501848 /loss/FRt4/detection_loss‚2.9679688175519305 1loss/SRqqt3/detection_loss‚1.7509112046385298 _runtime‚8750.423577943 1loss/SRqqt1/detection_loss‚1.8019569236785173 -loss/FBt1/detection_loss‚2.20383677312306²„©-_À!Ù 0loss/FBt4/assignment_loss‚0.0781949981335654 $ _timestamp‚1.788519124953305e+09 2loss/SRqqt1/assignment_loss‚1.2808993353694678 1loss/FBt1/assignment_loss‚0.23250522496865805 1loss/FBt2/assignment_loss‚0.10025641126070836 .loss/FRt3/detection_loss‚2.295122797501848 /loss/FRt4/detection_loss‚2.9679688175519305 1loss/SRqqt3/detection_loss‚1.7509112046385298 _runtime‚8750.423577943 1loss/SRqqt1/detection_loss‚1.8019569236785173 -loss/FBt1/detection_loss‚2.20383677312306 %loss/total_loss‚41.09645210566584 1loss/SRqqt4/detection_loss‚2.4620543365125305 /loss/FBt2/detection_loss‚0.5922910557443125 _step‚617 0loss/FRt4/assignment_loss‚2.4115000983079273 /loss/FBt4/detection_loss‚2.3565572930706873 /loss/FRt1/assignment_loss‚2.883172892942661 0loss/FRt3/assignment_loss‚2.8939358845670173 /loss/FRt2/detection_loss‚1.9232337770935817 epoch‚6 0loss/FRt2/assignment_loss‚1.7144326475950389 1loss/SRqqt3/assignment_loss‚1.229278185233778 0loss/SRqqt2/detection_loss‚1.188205051220587 1loss/FBt3/assignment_loss‚0.40830380421248264 .loss/FRt1/detection_loss‚2.261957817629392 /loss/FBt3/detection_loss‚3.4668107986450196 trainer/global_step‚7499 2loss/SRqqt2/assignment_loss‚0.9669220603269632 2loss/SRqqt4/assignment_loss‚1.6261439163375784é.ôW”Á!‚(:‰ ÞÅêÔáíý gpu.0.gpu‚35.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚39.0 gpu.0.powerWatts‚169.333' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.33325 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‘á*uÂ!‚(:ê ÞÅêÔˆïŠÿ! disk.sda.out‚1005.25927734375 disk.sdb1.out‚0 disk.sdb2.in‚3.1875 disk.sda1.in‚0 disk.md0p1.out‚ 968.85546875 disk.nvme0n1.in‚ 603.58984375# proc.memory.rssMB‚ 3000.48046875# disk.nvme1n1.out‚47941.12109375' proc.memory.availableMB‚ 1294.140625) proc.memory.percent‚18.31347942352295 proc.cpu.threads‚17) disk./.usagePercent‚75.85296523532215 disk.rbd0.in‚0 disk.sda1.out‚0 disk.dm-0.out‚97100.12890625 cpu‚25.211530365595852 disk.sda2.in‚ 6.08203125 disk.rbd0.out‚ 2.466796875 disk.sdb1.in‚0! disk.sdb.out‚1005.25927734375$ memory_percent‚92.10119247436523% disk./.usageGB‚2712.8694496154785" disk.sda2.out‚1005.25927734375 disk.nvme1n1.in‚ 498.80078125 disk.md0.out‚ 968.85546875 network.recv‚ 2.827662e+06" disk.sdb2.out‚1005.25927734375 disk.md0p1.in‚ 9.26953125" disk.nvme0n1.out‚ 49159.0078125 disk.sdb.in‚3.1875 disk.sda.in‚ 6.08203125 network.sent‚ 1.9191126e+07 disk.md0.in‚ 9.26953125 disk.rbd2.in‚0 disk.rbd2.out‚ 2.466796875 disk.dm-0.in‚ 1102.390625ó´V’Ã!‚(:‡ íÅêÔòÄ™ÿ gpu.0.gpu‚35.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚37.0 gpu.0.powerWatts‚123.712' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚30.928 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0nÊ¡hÄ!‚(:Ý íÅêÔœ§àÿ$ disk./.usageGB‚2712.869514465332 disk.nvme1n1.in‚ 498.80078125 disk.md0.in‚ 9.26953125 disk.rbd2.in‚0 disk.sdb2.in‚3.1875 proc.cpu.threads‚17 disk.md0p1.out‚ 970.515625 disk.sda.in‚ 6.08203125 disk.sda.out‚1006.994140625 disk.sda2.in‚ 6.08203125 disk.rbd0.in‚0 disk.sdb2.out‚1006.994140625 disk.dm-0.in‚ 1102.390625 disk.rbd0.out‚ 2.466796875) proc.memory.availableMB‚ 1294.38671875 disk.dm-0.out‚ 97244.2890625# disk.nvme0n1.out‚49302.83203125# disk.nvme1n1.out‚47941.45703125 disk.sdb1.in‚0 disk.md0.out‚ 970.515625 disk.nvme0n1.in‚ 603.58984375 disk.sdb.in‚3.1875 disk.rbd2.out‚ 2.466796875) disk./.usagePercent‚75.85296704855119 disk.md0p1.in‚ 9.26953125 disk.sdb.out‚1006.994140625 disk.sda1.out‚0# memory_percent‚92.0996904373169# proc.memory.rssMB‚ 3000.48046875 network.sent‚ 1.9298322e+07 disk.sda2.out‚1006.994140625 disk.sdb1.out‚0 disk.sda1.in‚0) proc.memory.percent‚18.31347942352295 cpu‚25.21119420211393 network.recv‚ 2.832262e+06i\^VžÅ!‚(:“ üÅêÔ“…“þ gpu.0.gpu‚34.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚115.634' gpu.0.enforcedPowerLimitWatts‚400.0) gpu.0.powerPercent‚28.908499999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0£ÛWlvÆ!‚(:ë üÅêԞʎÿ$ memory_percent‚92.10126399993896) proc.memory.percent‚18.31347942352295) disk./.usagePercent‚75.85471126823298 disk.md0.in‚ 9.26953125 disk.sda1.out‚0# disk.nvme0n1.out‚49374.71484375! disk.sdb.out‚1008.78759765625" disk.sda2.out‚1008.78759765625 network.recv‚ 2.836597e+06$ disk./.usageGB‚2712.931896209717 disk.sdb1.out‚0! disk.sda.out‚1008.78759765625 disk.rbd0.out‚ 2.466796875 disk.rbd2.in‚0 disk.rbd2.out‚ 2.466796875 disk.dm-0.in‚ 1102.390625 disk.dm-0.out‚97368.80078125 disk.md0p1.in‚ 9.26953125 disk.nvme0n1.in‚ 603.58984375" disk.sdb2.out‚1008.78759765625 disk.sdb.in‚3.1875 disk.sda.in‚ 6.08203125) proc.memory.availableMB‚ 1294.12890625# proc.memory.rssMB‚ 3000.48046875 cpu‚25.210895383425644 disk.rbd0.in‚0 disk.md0p1.out‚ 972.24609375 proc.cpu.threads‚17 disk.sda1.in‚0 disk.sda2.in‚ 6.08203125 disk.nvme1n1.in‚ 498.80078125 disk.sdb1.in‚0 disk.sdb2.in‚3.1875 network.sent‚ 1.9304268e+07" disk.nvme1n1.out‚ 47994.0859375 disk.md0.out‚ 972.24609375q«òÃ2Ç!- _wandbruntime‚8800  _runtime‚8800%³»ÇÈ!Á _runtime‚8800.927539544 _step‚618 & lr-AdamW/pg1‚0.0003187230163097474 & lr-AdamW/pg2‚0.0003187230163097474 trainer/global_step‚7549 % _timestamp‚1.7885191754570875e+09DpäÁÌÉ!Æ trainer/global_step‚7549 % _timestamp‚1.7885191754570875e+09 _runtime‚8800.927539544 _step‚618 & lr-AdamW/pg1‚0.0003187230163097474 & lr-AdamW/pg2‚0.0003187230163097474êó_¾½2Ê!- _wandbruntime‚8800  _runtime‚88001‘:Eü'Ë!öO QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚624 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 239345664 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚607584381744752 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 156014884 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 156014882 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177047040 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 142376469 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚621484077501952 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450780 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007241 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 345500986 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407872 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8588262073344 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1212 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26090 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8589161481216 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 189486104 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚45362670 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1187 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚630073256986112 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407872 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8965557438976 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8588262073344 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214665 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955591168 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 263285760 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26088 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚630073256986112 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 156014882 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚695340316238848 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚621484095504896 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 156014884 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚704305873677824 trainer/global_step‚7549 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 189486104 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚695340340178944 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚616141526898745 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8589161481216 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚616140612447965 _step‚619 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚630072339575296 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚704306136963584 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚607584364705392 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚626 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚621484077501952 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410816 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚97013173 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8556247742573 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚630072339575296 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411420 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 _runtime‚8800.928931301 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚621484095504896 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 142375843~¢C Ì!ûO R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚630072339575296 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411420 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 _runtime‚8800.928931301 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚621484095504896 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 142375843 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚624 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 239345664 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚607584381744752 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 156014884 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 156014882 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177047040 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 142376469 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚621484077501952 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450780 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007241 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 345500986 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407872 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8588262073344 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 H?DeviceStatsMonitor7s6’¾.on_train_batch_start/inactive_split.all.peak‚1212 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26090 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8589161481216 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 189486104 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚45362670 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1187 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚630073256986112 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407872 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8965557438976 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8588262073344 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214665 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955591168 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 263285760 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26088 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚630073256986112 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 156014882 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚695340316238848 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚621484095504896 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 156014884 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚704305873677824 trainer/global_step‚7549 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 189486104 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚695340340178944 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚616141526898745 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8589161481216 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚616140612447965 _step‚619 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚630072339575296 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚704306136963584 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚607584364705392 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚626 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚621484077501952 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410816 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚97013173 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8556247742573 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 JDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899421184 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 345545475 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 156034782 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8589374876672 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚621563725800448 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8557356403891 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 345571591 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚97025551 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8966948830720 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 345545475 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 345571591 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 142393889 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 242300928 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8589374876672 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 189536809 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚617 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚45368955 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 189510695 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 _runtime‚8801.903049251 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 156034780 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚616219701203411 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚704396608230400 trainer/global_step‚7549 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚607662344799520 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955591168 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚704396365929472 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚630153082674176 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚621563707797504 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955591168 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚704396365929472 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚630153082674176 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚621563707797504 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899421184 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 345545475 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 156034782 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8589374876672 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚621563725800448 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8557356403891 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 345571591 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚97025551 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8966948830720 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 345545475 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 345571591 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 142393889 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 242300928 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8589374876672 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 189536809 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚617 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚45368955 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 189510695 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 _runtime‚8801.903049251 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 156034780 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚616219701203411 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚704396608230400 trainer/global_step‚7549 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0ì'½GS2Ð!- _wandbruntime‚8801  _runtime‚8801àñ¿’\Ñ!Ö -loss/FBt3/detection_loss‚2.64199213385582 _runtime‚8801.904104926 0loss/FRt2/assignment_loss‚1.3671255445480346 /loss/FBt1/detection_loss‚2.5579178780317307 trainer/global_step‚7549 1loss/SRqqt3/detection_loss‚1.1362314768004835 _step‚621 /loss/FRt1/detection_loss‚2.1690087824804714 /loss/FRt2/detection_loss‚1.6532794645056128 1loss/SRqqt2/detection_loss‚0.9662367174831721 0loss/FBt2/detection_loss‚0.40385927479946987 /loss/FBt4/detection_loss‚2.5819336970647178 1loss/FBt4/assignment_loss‚0.09815335902688578 0loss/FRt3/assignment_loss‚1.8918311133980752 0loss/SRqqt3/assignment_loss‚1.17420939885472 .loss/FRt4/detection_loss‚2.990369207092694 epoch‚6 0loss/FRt1/assignment_loss‚2.6590480846027997 1loss/SRqqt1/detection_loss‚1.6140672558546065 %loss/total_loss‚38.50416178067129 % _timestamp‚1.7885191764339857e+09 0loss/FRt4/assignment_loss‚3.6158429043633595 2loss/SRqqt1/assignment_loss‚0.9948756481707096 2loss/SRqqt2/assignment_loss‚1.1055417725028327 0loss/FBt2/assignment_loss‚0.1358903088892262 0loss/FBt3/assignment_loss‚0.4252528761358917 1loss/SRqqt4/detection_loss‚2.5463776723031075 2loss/SRqqt4/assignment_loss‚1.5614993879871983 1loss/FBt1/assignment_loss‚0.05215851026301266 .loss/FRt3/detection_loss‚2.161459311656654_U¿aÒ!Û 0loss/FRt4/assignment_loss‚3.6158429043633595 2loss/SRqqt1/assignment_loss‚0.9948756481707096 2loss/SRqqt2/assignment_loss‚1.1055417725028327 0loss/FBt2/assignment_loss‚0.1358903088892262 0loss/FBt3/assignment_loss‚0.4252528761358917 1loss/SRqqt4/detection_loss‚2.5463776723031075 2loss/SRqqt4/assignment_loss‚1.5614993879871983 1loss/FBt1/assignment_loss‚0.05215851026301266 .loss/FRt3/detection_loss‚2.161459311656654 -loss/FBt3/detection_loss‚2.64199213385582 _runtime‚8801.904104926 0loss/FRt2/assignment_loss‚1.3671255445480346 /loss/FBt1/detection_loss‚2.5579178780317307 trainer/global_step‚7549 1loss/SRqqt3/detection_loss‚1.1362314768004835 _step‚621 /loss/FRt1/detection_loss‚2.1690087824804714 /loss/FRt2/detection_loss‚1.6532794645056128 1loss/SRqqt2/detection_loss‚0.9662367174831721 0loss/FBt2/detection_loss‚0.40385927479946987 /loss/FBt4/detection_loss‚2.5819336970647178 1loss/FBt4/assignment_loss‚0.09815335902688578 0loss/FRt3/assignment_loss‚1.8918311133980752 0loss/SRqqt3/assignment_loss‚1.17420939885472 .loss/FRt4/detection_loss‚2.990369207092694 epoch‚6 0loss/FRt1/assignment_loss‚2.6590480846027997 1loss/SRqqt1/detection_loss‚1.6140672558546065 %loss/total_loss‚38.50416178067129 % _timestamp‚1.7885191764339857e+09íÿæÑ’Ó!‚(:‡ ‹ÆêÔ§è´þ gpu.0.gpu‚40.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚37.0 gpu.0.powerWatts‚114.408' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.602 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0n\*nÔ!‚(:ã ‹ÆêÔŸËôþ disk.dm-0.out‚97441.21484375" disk.nvme0n1.out‚ 49446.8671875" disk.sda2.out‚1010.89697265625 disk.sda1.out‚0 disk.sdb.in‚3.1875$ memory_percent‚92.10138320922852% disk./.usageGB‚2712.9947204589844 disk.nvme1n1.in‚ 498.80078125 disk.sda1.in‚0' proc.memory.availableMB‚ 1294.109375 proc.cpu.threads‚17( disk./.usagePercent‚75.8564678605365 disk.nvme0n1.in‚ 603.58984375 disk.md0.out‚ 974.28125 disk.dm-0.in‚ 1102.390625! disk.sdb.out‚1010.89697265625 disk.md0p1.in‚ 9.26953125 network.sent‚ 1.9310167e+07 disk.sda2.in‚ 6.08203125 disk.rbd0.in‚0 disk.rbd0.out‚ 2.466796875# disk.nvme1n1.out‚47994.34765625 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.md0.in‚ 9.26953125 disk.rbd2.in‚0 disk.rbd2.out‚ 2.466796875" disk.sdb2.out‚1010.89697265625 network.recv‚ 2.840667e+06 disk.sda.in‚ 6.08203125! disk.sda.out‚1010.89697265625 disk.sdb2.in‚3.1875 disk.md0p1.out‚ 974.28125) proc.memory.percent‚18.31347942352295 cpu‚25.210595007646635# proc.memory.rssMB‚ 3000.48046875˜¦¼D”Õ!‚(:‰ šÆêÔ…–þ gpu.0.gpu‚40.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚168.715' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.17875 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0SÎ…/Ö!‚(:å šÆêÔê¿Œÿ! disk.sda.out‚101QÂ|ðA2.52197265625" disk.nvme1n1.out‚ 48046.9609375# memory_percent‚92.1013593673706) proc.memory.percent‚18.31347942352295% disk./.usageGB‚2713.1199111938477" disk.sdb2.out‚1012.52197265625 disk.dm-0.in‚ 1102.390625! disk.sdb.out‚1012.52197265625 disk.rbd0.in‚0 cpu‚25.210265004518703 disk.rbd2.in‚0 disk.dm-0.out‚97529.80078125 disk.md0p1.in‚ 9.26953125 disk.sda.in‚ 6.08203125" disk.sda2.out‚1012.52197265625 disk.rbd0.out‚ 2.466796875) disk./.usagePercent‚75.85996824587966 disk.sda1.out‚0# disk.nvme0n1.out‚49482.83984375# proc.memory.rssMB‚ 3000.48046875 network.recv‚ 2.846824e+06 disk.md0.in‚ 9.26953125 disk.rbd2.out‚ 2.466796875 disk.nvme0n1.in‚ 603.58984375 disk.sda2.in‚ 6.08203125 disk.nvme1n1.in‚ 498.80078125 network.sent‚ 1.9418141e+07 disk.sdb2.in‚3.1875) proc.memory.availableMB‚ 1294.11328125 disk.sdb1.out‚0 disk.md0.out‚ 975.84375 disk.sda1.in‚0 disk.md0p1.out‚ 975.84375 disk.sdb.in‚3.1875 proc.cpu.threads‚17 disk.sdb1.in‚0…̈“×!‚(:ˆ ©ÆêÔ €“þ gpu.0.gpu‚14.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚167.774' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚41.9435 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0—B9=tØ!‚(:é ©ÆêÔø¤–ÿ) proc.memory.percent‚18.31347942352295 disk.rbd2.out‚ 2.466796875 disk.nvme0n1.in‚ 603.58984375" disk.nvme0n1.out‚ 49554.8046875! disk.sdb.out‚1014.03662109375 proc.cpu.threads‚17 disk.rbd2.in‚0 disk.rbd0.in‚0 disk.sdb1.out‚0$ memory_percent‚92.10000038146973" disk.sdb2.out‚1014.03662109375( proc.memory.availableMB‚ 1294.3359375 disk.md0.in‚ 9.26953125 disk.sdb2.in‚3.1875 disk.sda1.in‚0 disk.dm-0.out‚97654.55078125 disk.md0p1.in‚ 9.26953125 disk.sda2.in‚ 6.08203125" disk.sda2.out‚1014.03662109375 network.recv‚ 2.850196e+06 disk.sda1.out‚0 disk.md0p1.out‚ 977.30078125# proc.memory.rssMB‚ 3000.48046875 cpu‚25.209965010177804 disk.dm-0.in‚ 1102.390625 disk.sda.in‚ 6.08203125# disk.nvme1n1.out‚48099.74609375 network.sent‚ 1.9423439e+07( disk./.usagePercent‚75.8599554466158$ disk./.usageGB‚2713.119453430176 disk.md0.out‚ 977.30078125! disk.sda.out‚1014.03662109375 disk.nvme1n1.in‚ 498.80078125 disk.sdb.in‚3.1875 disk.rbd0.out‚ 2.466796875 disk.sdb1.in‚0Í‚‘Å’Ù!‚(:‡ ¸ÆêÔâæìý gpu.0.gpu‚53.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚167.604' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚41.901 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0V:önÚ!‚(:ã ¸ÆêÔáà¿ÿ! disk.sdb.out‚1015.71728515625! disk.sda.out‚1015.71728515625 disk.sda2.in‚ 6.08203125 proc.cpu.threads‚17 disk.sda1.out‚0 disk.dm-0.out‚ 97762.6953125" disk.nvme1n1.out‚ 48099.9453125 disk.sdb1.out‚0) disk./.usagePercent‚75.85216026828623 disk.md0p1.out‚ 978.91015625 disk.rbd0.out‚ 2.466796875 disk.nvme1n1.in‚ 498.80078125 disk.rbd2.out‚ 2.466796875( proc.memory.availableMB‚ 1293.9765625$ disk./.usageGB‚2712.840660095215 disk.dm-0.in‚ 1102.390625 disk.nvme0n1.out‚49662.75 disk.sdb2.in‚3.1875" disk.sdb2.out‚1015.71728515625 disk.rbd0.in‚0 disk.md0.out‚ 978.91015625 disk.rbd2.in‚0$ memory_percent‚92.10219383239746# proc.memory.rssMB‚ 3000.48046875 disk.md0p1.in‚ 9.26953125" disk.sda2.out‚1015.71728515625 disk.sdb1.in‚0 network.recv‚ 2.854828e+06 disk.sda1.in‚0 disk.nvme0n1.in‚ 603.58984375 disk.sdb.in‚3.1875 disk.sda.in‚ 6.08203125) proc.memory.percent‚18.31347942352295 cpu‚25.209664639961392 network.sent‚ 1.9429445e+07 disk.md0.in‚ 9.26953125]ÈÍ~2Û!- _wandbruntime‚8852  _runtime‚8852ºýÝ!ÉÜ!à trainer/global_step‚7599 % _timestamp‚1.7885192271192741e+09 _runtime‚8852.589665426 _step‚622 ' lr-AdamW/pg1‚0.00031275136689217295 ' lr-AdamW/pg2‚0.00031275136689217295¥<à[ÎÝ!È % _timestamp‚1.7885192271192741e+09 _runtime‚8852.589665426 _step‚622 ' lr-AdamW/pg1‚0.00031275136689217295 ' lr-AdamW/pg2‚0.00031275136689217295 trainer/global_step‚7599îÁQf¡2Þ!- _wandbruntime‚8852  _runtime‚8852 XÃü'ß!öO WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8643916653568 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 143281297 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚97631973 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411420 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚634110426343936 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917410816 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 190745094 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1212 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410816 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8644816061440 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚97631975 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007241 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 190745094 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚625465592279552 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 157009784 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9023512321024 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214665 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚45649324 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8611694399681 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 trainer/global_step‚7599 % _timestamp‚1.7885192271204484e+09 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 190719006 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407872 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 143281914 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚617 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚708830044493824 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26090 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 347754878 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚708829789596672 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 347728788 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚634109508933120 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚699806277275648 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚611483386451152 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚611483369411792 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 157009784 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚634110426343936 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚625465610282496 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 399863808 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚699806301215744 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚625465592279552 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177047040 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚625465610282496 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8644816061440 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚97631975 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007241 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 190745094 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚625465592279552 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 157009784 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9023512321024 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214665 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚45649324 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8611694399681 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 trainer/global_step‚7599 % _timestamp‚1.7885192271204484e+09 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 190719006 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407872 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 143281914 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚617 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚708830044493824 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26090 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 347754878 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚708829789596672 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 347728788 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚634109508933120 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚699806277275648 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚611483386451152 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚611483369411792 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 157009784 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚634110426343936 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚625465610282496 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 399863808 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚699806301215744 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚625465592279552 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177047040 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚625465610282496 JDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26114 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 347773277 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9024661076992 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚611561349505920 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 347773277 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 157029680 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚634190252032000 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 190769711 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26114 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚634191169456128 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1187 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 261175296 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 143299455 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 157029682 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899421184 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 347799393 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899421184 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚634191169456128 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚625545222575104 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 190743597 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚625545240578048 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8645928878080 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8645029456896 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚45655727 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177047040 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚625545240578048 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 190769711 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8645928878080 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚699895620436480 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 " _timestamp‚1.7885192280997e+09 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 347799393 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚45655106 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚708920257573376 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 237235200 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8612803060999 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8645029456896 C=DeviceStatsMonitor.on_train_ba*×Û Ð tch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚620174152566919 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚624 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 " _timestamp‚1.7885192280997e+09 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 347799393 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚45655106 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚708920257573376 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 237235200 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8612803060999 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8645029456896 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚620174152566919 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚624 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26114 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 347773277 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9024661076992 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚611561349505920 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 347773277 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 157029680 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚634190252032000 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 190769711 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26114 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚634191169456128 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1187 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 261175296 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 143299455 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 157029682 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899421184 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 347799393 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899421184 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚634191169456128 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚625545222575104 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 190743597 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚625545240578048 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8645928878080 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8645029456896 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚45655727 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177047040 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚625545240578048 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 190769711 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8645928878080 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚699895620436480ð® ‘¥2ä!- _wandbruntime‚8853  _runtime‚8853öÒ_å!Ù /loss/FRt4/assignment_loss‚2.589772873911364 2loss/SRqqt3/assignment_loss‚1.1558454128211004 .loss/FRt1/detection_loss‚2.042611845697528 /loss/FRt3/detection_loss‚1.9419192437981736 1loss/SRqqt4/detection_loss‚2.5407009798547495 1loss/FBt3/assignment_loss‚0.17943082896194287 /loss/FRt4/detection_loss‚2.9963586782586984 2loss/SRqqt4/assignment_loss‚1.8891111502180928 1loss/FBt4/assignment_loss‚0.12422161001913165 /loss/FRt2/detection_loss‚1.3891131472080311 1loss/SRqqt3/detection_loss‚1.3819011658952947 /loss/FBt3/detection_loss‚2.2671234352248057 % _timestamp‚1.7885192281017623e+09 /loss/FRt1/assignment_loss‚2.971320819641863 2loss/SRqqt1/assignment_loss‚1.3140538421659065 2loss/SRqqt2/assignment_loss‚0.8351502173007971 /loss/FBt1/detection_loss‚3.3894734025001525 trainer/global_step‚7599 1loss/FBt1/assignment_loss‚0.17405124631040964 0loss/FBt2/assignment_loss‚0.1470099568708694 epoch‚6 0loss/FRt2/assignment_loss‚1.4083061953808398 0loss/FRt3/assignment_loss‚2.1459324092700562 /loss/FBt2/detection_loss‚0.4756400304908554 .loss/FBt4/detection_loss‚2.192081023346294 &loss/total_loss‚38.027172822217096 _runtime‚8853.571928793 _step‚625 1loss/SRqqt1/detection_loss‚1.6994324275957686 1loss/SRqqt2/detection_loss‚0.7766108794743756(é!dæ!Þ 1loss/FBt3/assignment_loss‚0.17943082896194287 /loss/FRt4/detection_loss‚2.9963586782586984 2loss/SRqqt4/assignment_loss‚1.8891111502180928 1loss/FBt4/assignment_loss‚0.12422161001913165 /loss/FRt2/detection_loss‚1.3891131472080311 1loss/SRqqt3/detection_loss‚1.3819011658952947 /loss/FBt3/detection_loss‚2.2671234352248057 % _timestamp‚1.7885192281017623e+09 /loss/FRt1/assignment_loss‚2.971320819641863 2loss/SRqqt1/assignment_loss‚1.3140538421659065 2loss/SRqqt2/assignment_loss‚0.8351502173007971 /loss/FBt1/detection_loss‚3.3894734025001525 trainer/global_step‚7599 1loss/FBt1/assignment_loss‚0.17405124631040964 0loss/FBt2/assignment_loss‚0.1470099568708694 epoch‚6 0loss/FRt2/assignment_loss‚1.4083061953808398 0loss/FRt3/assignment_loss‚2.1459324092700562 /loss/FBt2/detection_loss‚0.4756400304908554 .loss/FBt4/detection_loss‚2.192081023346294 &loss/total_loss‚38.027172822217096 _runtime‚8853.571928793 _step‚625 1loss/SRqqt1/detection_loss‚1.6994324275957686 1loss/SRqqt2/detection_loss‚0.7766108794743756 /loss/FRt4/assignment_loss‚2.589772873911364 2loss/SRqqt3/assignment_loss‚1.1558454128211004 .loss/FRt1/detection_loss‚2.042611845697528 /loss/FRt3/detection_loss‚1.9419192437981736 1loss/SRqqt4/detection_loss‚2.5407009798547495ñ烒ç!‚(:‡ ÇÆêÔ±Û“ÿ gpu.0.gpu‚55.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚166.988' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚41.747 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0KÌÂ\è!‚(:Ñ ÇÆêÔ³Œ¼ÿ disk.rbd0.in‚0 disk.rbd0.out‚ 2.466796875 disk.sdb1.in‚0 disk.sdb.in‚3.1875" disk.sda2.out‚1017.80712890625 disk.nvme1n1.in‚ 498.80078125 disk.rbd2.out‚ 2.466796875$ disk./.usageGB‚2712.839385986328# proc.memory.rssMB‚ 3000.48046875 disk.md0p1.in‚ 9.26953125 disk.md0p1.out‚980.9375) proc.memory.availableMB‚ 1293.68359375) disk./.usagePercent‚75.85212464366849 disk.dm-0.out‚98033.25 disk.md0.in‚ 9.26953125 disk.sdb2.in‚3.1875 disk.dm-0.in‚ 1102.390625 disk.sdb1.out‚0 disk.md0.out‚980.9375$ memory_percent‚92.10398197174072 proc.cpu.threads‚17 disk.nvme0n1.in‚ 603.58984375 disk.sda2.in‚ 6.08203125 disk.nvme1n1.out‚48309.75" disk.sdb2.out‚1017.80712890625) proc.memory.percent‚18.31347942352295 cpu‚25.209367223729306 disk.sda1.in‚0 disk.nvme0n1.out‚49723.5! disk.sdb.out‚1017.80712890625 disk.sda.in‚ 6.08203125! disk.sda.out‚1017.80712890625 disk.rbd2.in‚0 network.sent‚ 1.9435361e+07 network.recv‚ 2.858762e+06 disk.sda1.out‚0)ñ”é!‚(:‰ ÖÆêԦݲÿ gpu.0.gpu‚48.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚143.263' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚35.81575 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0»ùÌ_hê!‚(:Ý ÖÆêÔòÑ€) disk./.usagePercent‚75.86441183030838$ disk./.usageGB‚2713.278835296631 disk.rbd2.in‚0 network.recv‚ 2.864183e+06 disk.sdb2.in‚3.1875 disk.sda1.in‚0 disk.md0p1.in‚ 9.26953125 disk.nvme0n1.in‚ 603.58984375 disk.nvme1n1.out‚ 48413.96875 disk.sda1.out‚0 disk.md0p1.out‚ 982.40625 disk.sda2.in‚ 6.08203125 disk.rbd0.in‚0 network.sent‚ 1.9542645e+07 disk.sdb.in‚3.1875 disk.sdb.out‚1019.3427734375 disk.sda.in‚ 6.08203125 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.md0.out‚ 982.40625! disk.sdb2.out‚1019.3427734375 disk.dm-0.out‚ 98138.2109375! disk.sda2.out‚1019.3427734375 disk.nvme1n1.in‚ 498.80078125 disk.md0.in‚ 9.26953125 disk.dm-0.in‚ 1102.390625 disk.rbd0.out‚ 2.466796875 disk.rbd2.out‚ 2.466796875) proc.memory.percent‚18.31347942352295" disk.nvme0n1.out‚ 49724.2421875 disk.sda.out‚1019.3427734375$ memory_percent‚92.09940433502197) proc.memory.availableMB‚ 1294.43359375# proc.memory.rssMB‚ 3000.48046875 cpu‚25.209091915801245 proc.cpu.threads‚17lŒ¹Ôvë!‚(:ë åÆêÔŠÿì disk.rbd2.in‚0" disk.sdb2.out‚1020.94091796875 disk.rbd0.out‚ 2.466796875 disk.md0.in‚ 9.26953125 disk.md0.out‚ 983.94921875 disk.nvme0n1.in‚ 618.11328125) proc.memory.percent‚18.31347942352295) disk./.usagePercent‚75.86686502254719 disk.sdb1.in‚0 disk.rbd2.out‚ 2.466796875 disk.sda1.in‚0 disk.dm-0.in‚ 1116.9140625 disk.md0p1.out‚ 983.94921875 cpu‚25.20878922480721 disk.sdb2.in‚3.1875 disk.md0p1.in‚ 9.26953125( proc.memory.availableMB‚ 1294.3203125 disk.nvme1n1.in‚ 498.80078125# disk.nvme0n1.out‚49816.40234375$ memory_percent‚92.10009574890137 network.sent‚ 1.9547884e+07" disk.sda2.out‚1020.94091796875 disk.rbd0.in‚0# disk.nvme1n1.out‚48467.68359375 disk.sdb1.out‚0 disk.sda1.out‚0! disk.sdb.out‚1020.94091796875 network.recv‚ 2.867429e+06% disk./.usageGB‚2713.3665733337402 disk.sda.in‚ 6.08203125! disk.sda.out‚1020.94091796875 disk.sda2.in‚ 6.08203125 disk.dm-0.out‚ 98284.0859375# proc.memory.rssMB‚ 3000.48046875 proc.cpu.threads‚17 disk.sdb.in‚3.18754¦ÓK”ì!‚(:‰ åÆêÔ©ñ¾ª gpu.0.gpu‚45.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚37.0 gpu.0.powerWatts‚166.825' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚41.70625 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Äp0È2í!- _wandbruntime‚8908  _runtime‚8908B™ìÈî! ' lr-AdamW/pg1‚0.00030680391231828697 ' lr-AdamW/pg2‚0.00030680391231828697 trainer/global_step‚7649 $ _timestamp‚1.788519283502353e+09 _runtime‚8908.972932167 _step‚626y@àÍï!Ç ' lr-AdamW/pg2‚0.00030680391231828697 trainer/global_step‚7649 $ _timestamp‚1.788519283502353e+09 _runtime‚8908.972932167 _step‚626 ' lr-AdamW/pg1‚0.00030680391231828697ò;XÃ.2ð!- _wandbruntime‚8908  _runtime‚8908Šªxü'ñ!öO M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 191951908 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚638146678290944 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 158004684 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 244411392 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 _step‚627 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚638147595701760 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8700470641664 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1212 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚98250775 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9081505179648 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1187 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 trainer/global_step‚7649 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177047040 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 144185577 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8668038468209 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955591168 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8699571233792 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚629447107057152 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8699571233792 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚713353987903488 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚638147595701760 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 191951908 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 158004684 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 191977996 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 399863808 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 349956590 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407872 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚98250773 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007241 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410816 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450780 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚604 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 _runtime‚8908.974399243 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 158004682 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26088 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚45935406 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚629447107057152 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 158004682 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26090 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 144186181 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚45934804 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚615382374118192 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411420 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8667141056789Žƒ>·Óò!ûO M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 191951908 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚638146678290944 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 158004684 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 244411392 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 _step‚627 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚638147595701760 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8700470641664 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1212 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚98250775 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9081505179648 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1187 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 trainer/global_step‚7649 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177047040 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 144185577 60Dì8þ2. eviceStatsMonitor.on_train_batch_start/num_ooms‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8668038468209 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955591168 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8699571233792 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚629447107057152 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8699571233792 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚713353987903488 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚638147595701760 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 191951908 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 158004684 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 191977996 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 399863808 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 349956590 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407872 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚98250773 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007241 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410816 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450780 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚604 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 _runtime‚8908.974399243 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 158004682 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26088 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚45935406 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚629447107057152 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 158004682 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26090 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 144186181 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚45934804 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚615382374118192 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411420 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8667141056789óÛq2ó!- _wandbruntime‚8910  _runtime‚8910ÑP¶h'ô!€N TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 158024580 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚98263151 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8701583458304 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177047040 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚638228338813952 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26114 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 144203652 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 158024582 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚629526737352704 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8700684037120 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8669147129735 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚638227421389824 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚629526755355648 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8701583458304 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8700684037120 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 191976499 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450988 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 350027195 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 192002613 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚615460371251680 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411628 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31686 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0[˜°˜ 'õ!…N YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177047040 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 158024580 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚629526737352704 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚704361557533184 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 220457984 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31686 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 158024580 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚98263151 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8701583458304 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177047040 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚638228338813952 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26114 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 144203652 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 158024582 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚629526737352704 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8700684037120 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8669147129735 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚638227421389824 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚629526755355648 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8701583458304 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8700684037120 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 191976499 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450988 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 350027195 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 192002613 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚615460371251680 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411628 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 FDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8723485125317 trainer/global_step‚7699 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411420 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26088 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917410816 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚642183847648768 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚708738199349248 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9139728213504 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 _runtime‚8967.460466018 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚46221470 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 399863808 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚628004880989269 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 158999584 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407872 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 158999582 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955591168 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8756125221888 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407872 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚98869575 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚609 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1212 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚46220863 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚633428639837696 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 158999584 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26088 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 % _timestamp‚1.7885193419896238e+09 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 193210898 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177047040 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚98869573 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 158999582 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214665 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 352210482 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚633428621834752 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚717877951502848 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 193184810 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚633428621834752 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1187 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 237248512 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚619281395863952 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0ýË((†"ûO LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚633428639837696 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 158999584 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26088 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 % _timestamp‚1.7885193419896238e+09 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 193210898 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177047040 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚98869573 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 158999582 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214665 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 352210482 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚633428621834752 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚717877951502848 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 193184810 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚633428621834752 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1187 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 237248512 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚619281395863952 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 352184392 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 193184810 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8755225814016 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8723485125317 trainer/global_step‚7699 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411420 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26088 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917410816 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚642183847648768 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚708738199349248 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9139728213504 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 _runtime‚8967.460466018 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚46221470 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 399863808 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚628004880989269 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 158999584 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407872 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 158999582 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955591168 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8756125221888 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407872 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚98869575 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚609 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1212 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚46220863÷Á¯2‡"- _wandbruntime‚8968  _runtime‚8968C—t­Ëˆ"€N SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899421184 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 159019482 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9140871707136 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚642264590747648 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 193209401 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8757238038528 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8756338617344 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177047040 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚633 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚619359358918720 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚98881951 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 399863808 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8757238038528 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 145108463 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚46226514 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚619359375958080 =7DeviceStatsMonitop)âü; r.on_train_batch_end/num_alloc_retries‚0 trainer/global_step‚7699 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 159019480 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚708827542510080 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 159019480 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 256980992 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 % _timestamp‚1.7885193429953716e+09 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 352254997 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚633508252130304 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 145109096 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚46227145 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 233040896 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9140638666240 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450988 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007241 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 352228881 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 159019482 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚631 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚633508270133248RqOf '‰"…N XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26114 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚642265508171776 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚642265508171776 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26114 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955591168 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚633508252130304 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚717968414217216 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚619359358918720 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚98881951 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 399863808 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8757238038528 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 145108463 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚46226514 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚619359375958080 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 trainer/global_step‚7699 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 159019480 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚708827542510080 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 159019480 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 256980992 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 % _timestamp‚1.7885193429953716e+09 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 352254997 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚633508252130304 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 145109096 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚46227145 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 233040896 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9140638666240 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450988 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007241 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 352228881 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 159019482 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚631 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚633508270133248 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899421184 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 159019482 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9140871707136 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚642264590747648 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 193209401 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8757238038528 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8756338617344 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177047040 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚633 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 PDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407872 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8810880394240 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚713204160386048 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214665 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9197848697344 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1212 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 205791232 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411420 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 159994484 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚646221934417408 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955591168 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚623180400570352 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26090 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 194443800 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410816 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚646221017006592 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407872 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8810880394240 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚713204160386048 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214665 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9197848697344 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1212 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 205791232 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411420 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 159994484 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚646221934417408 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955591168 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚623180400570352 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26090 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 194443800 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410816 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚646221017006592 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 JDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 146015100 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 399863808 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8779143032323 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26114 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 354482799 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚632037506657443 _runtime‚9020.008080838 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177047040 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚623258380664480 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8780040443951 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 194442303 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚646302677529600 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899421184 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚722492316675072 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9199042846208 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8811993197568 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 194442303 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 354482799 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 354456683 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31686 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚99500751 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚722492546393088 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955591168 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚623258363625120 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177047040 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899421184 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚617 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚615 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9198837068288 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 146015717 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1187 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚46514351 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 229718016 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 160014382 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚646302677529600 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚99500749 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450988 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8811993197568¥8y '"…N XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8811993197568 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 194442303 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 354482799 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 354456683 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31686 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚99500751 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚722492546393088 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955591168 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚623258363625120 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177047040 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899421184 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚617 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚615 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9198837068288 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 146015717 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1187 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚46514351 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 229718016 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 160014382 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚646302677529600 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚99500749 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450988 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8811993197568 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 146015100 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 399863808 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8779143032323 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26114 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 354482799 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚632037506657443 _runtime‚9020.008080838 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177047040 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚623258380664480 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8780040443951 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 194442303 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚646302677529600 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899421184 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚722492316675072 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9199042846208ü¤â!~2ž"- _wandbruntime‚9020  _runtime‚9020=‘ \Ÿ"Ö 1loss/SRqqt4/detection_loss‚2.3873595972855886 /loss/FRt3/assignment_loss‚2.561516489301409 .loss/FRt3/detection_loss‚1.991618744230696 epoch‚6 _runtime‚9020.009987507 _step‚637 2loss/SRqqt1/assignment_loss‚1.0487053780100093 0loss/FBt2/detection_loss‚0.41536673085850684 .loss/FBt4/detection_loss‚2.577122449874878 1loss/FBt1/assignment_loss‚0.03804683673644149 0loss/FBt2/assignment_loss‚0.0640307059604911 1loss/SRqqt2/detection_loss‚1.1229613322404124 /loss/FRt1/assignment_loss‚2.946575335093907 0loss/FRt2/assignment_loss‚1.5422068754067788 2loss/SRqqt3/assignment_loss‚1.4047462212158657 1loss/FBt3/assignment_loss‚0.06188048981130123 .loss/FRt1/detection_loss‚2.445482966800619 /loss/FRt4/detection_loss‚2.8326493707196465 1loss/SRqqt2/assignment_loss‚1.034108259016648 /loss/FRt2/detection_loss‚1.7897975459479942 &loss/total_loss‚39.509399331258756 1loss/SRqqt1/detection_loss‚1.5990800608952458 .loss/FBt1/detection_loss‚2.746323376893997 /loss/FRt4/assignment_loss‚3.234766195560324 1loss/FBt4/assignment_loss‚0.04094003875838502 1loss/SRqqt3/detection_loss‚1.4428023611805563 /loss/FBt3/detection_loss‚2.8993059396743774 % _timestamp‚1.7885193945398302e+09 trainer/global_step‚7749 2loss/SRqqt4/assignment_loss‚1.2820060297846794ªñ7 W "Û 1loss/SRqqt2/assignment_loss‚1.034108259016648 /loss/FRt2/detection_loss‚1.7897975459479942 &loss/total_loss‚39.509399331258756 1loss/SRqqt1/detection_loss‚1.5990800608952458 .loss/FBt1/detection_loss‚2.746323376893997 /loss/FRt4/assignment_loss‚3.234766195560324 1loss/FBt4/assignment_loss‚0.04094003875838502 1loss/SRqqt3/detection_loss‚1.4428023611805563 /loss/FBt3/detection_loss‚2.8993059396743774 % _timestamp‚1.7885193945398302e+09 trainer/global_step‚7749 2loss/SRqqt4/assignment_loss‚1.2820060297846794 1loss/SRqqt4/detection_loss‚2.3873595972855886 /loss/FRt3/assignment_loss‚2.561516489301409 .loss/FRt3/detection_loss‚1.991618744230696 epoch‚6 _runtime‚9020.009987507 _step‚637 2loss/SRqqt1/assignment_loss‚1.0487053780100093 0loss/FBt2/detection_loss‚0.41536673085850684 .loss/FBt4/detection_loss‚2.577122449874878 1loss/FBt1/assignment_loss‚0.03804683673644149 0loss/FBt2/assignment_loss‚0.0640307059604911 1loss/SRqqt2/detection_loss‚1.1229613322404124 /loss/FRt1/assignment_loss‚2.9465]ÜëZ 75335093907 0loss/FRt2/assignment_loss‚1.5422068754067788 2loss/SRqqt3/assignment_loss‚1.4047462212158657 1loss/FBt3/assignment_loss‚0.06188048981130123 .loss/FRt1/detection_loss‚2.445482966800619 /loss/FRt4/detection_loss‚2.8326493707196465ý[3ÙÃ’¡"‚(:‡ ìÇêÔ¥ÀÌþ gpu.0.gpu‚22.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚175.844' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.961 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ѽ­$r¢"‚(:ç ìÇêÔɼƒÿ disk.rbd0.out‚ 2.466796875 disk.nvme1n1.in‚ 498.80078125% disk./.usageGB‚2712.7249717712402# disk.nvme1n1.out‚49461.47265625 disk.sdb1.in‚0 disk.md0.out‚ 999.76953125) proc.memory.availableMB‚ 1293.83984375# proc.memory.rssMB‚ 3000.48046875! disk.sdb2.out‚1037.3525390625 disk.sda1.out‚0 disk.nvme0n1.in‚ 618.11328125 disk.sda.out‚1037.3525390625 network.sent‚ 1.9884204e+07" disk.nvme0n1.out‚ 50251.6796875$ memory_percent‚92.10302829742432 disk.md0.in‚ 9.26953125) disk./.usagePercent‚75.84892557432856 disk.dm-0.out‚99713.15234375 disk.md0p1.in‚ 9.26953125 disk.sda.in‚ 6.08203125 disk.sda2.in‚ 6.08203125 network.recv‚ 2.908743e+06 disk.sdb2.in‚3.1875 disk.sdb.out‚1037.3525390625 cpu‚25.2060794205317 disk.rbd0.in‚0 disk.sdb1.out‚0 disk.rbd2.in‚0 disk.rbd2.out‚ 2.466796875) proc.memory.percent‚18.31347942352295 proc.cpu.threads‚17 disk.md0p1.out‚ 999.76953125 disk.sdb.in‚3.1875 disk.sda1.in‚0 disk.dm-0.in‚ 1116.9140625! disk.sda2.out‚1037.3525390625ÒâQV”£"‚(:‰ ûÇêÔŠ¨ïý gpu.0.gpu‚56.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚39.0 gpu.0.powerWatts‚172.727' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.18175 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‡~dIv¤"‚(:ë ûÇêÔ³°íþ disk.md0.out‚ 1001.52734375# proc.memory.rssMB‚ 3000.48046875) proc.memory.percent‚18.31347942352295) disk./.usagePercent‚75.85088172848768" disk.sdb2.out‚1039.16748046875 disk.nvme1n1.in‚ 498.80078125 disk.sda1.in‚0 disk.rbd2.out‚ 2.466796875 disk.sda1.out‚0 disk.nvme0n1.in‚ 618.11328125" disk.nvme0n1.out‚ 50403.6953125 disk.sda2.in‚ 6.08203125 disk.rbd0.in‚0 disk.sdb1.in‚0 network.sent‚ 1.999176e+07 disk.md0p1.in‚ 9.26953125 disk.rbd0.out‚ 2.466796875 disk.md0.in‚ 9.26953125) proc.memory.availableMB‚ 1293.92578125 cpu‚25.20576718981856 disk.dm-0.in‚ 1116.9140625 disk.dm-0.out‚99865.28515625 disk.sdb.in‚3.1875! disk.sda.out‚1039.16748046875 disk.rbd2.in‚0 network.recv‚ 2.913904e+06 disk.sdb2.in‚3.1875" disk.sda2.out‚1039.16748046875 disk.sdb1.out‚0# memory_percent‚92.1025037765503$ disk./.usageGB‚2712.794933319092! disk.sdb.out‚1039.16748046875 proc.cpu.threads‚17 disk.md0p1.out‚ 1001.52734375 disk.sda.in‚ 6.08203125# disk.nvme1n1.out‚49461.58984375ú'/ ”¥"‚(:‰ ŠÈêÔü†ßý gpu.0.gpu‚49.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚159.419' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚39.85475 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÌrÐw¦"‚(:ì ŠÈêÔ€Ó²ÿ disk.md0.in‚ 9.26953125 disk.sda1.out‚0 disk.md0p1.in‚ 9.26953125 disk.nvme0n1.in‚ 618.11328125 disk.md0.out‚ 1003.0078125# proc.memory.rssMB‚ 3000.48046875 disk.sdb1.out‚0 disk.sda.in‚ 6.08203125! disk.sda.out‚1040.69873046875 disk.sdb1.in‚0" disk.sdb2.out‚1040.69873046875$ memory_percent‚92.10512638092041) proc.memory.percent‚18.31347942352295 disk.sda2.in‚ 6.08203125 disk.rbd0.out‚ 2.466796875 disk.rbd2.out‚ 2.466796875# disk.nvme0n1.out‚50467.58984375! disk.sdb.out‚1040.69873046875 proc.cpu.threads‚17 disk.rbd0.in‚0! disk.nvme1n1.out‚ 49461.953125 disk.rbd2.in‚0 disk.dm-0.in‚ 1116.9140625 disk.sdb.in‚3.1875 network.sent‚ 1.9998044e+07 disk.nvme1n1.in‚ 498.80078125 disk.dm-0.out‚99929.54296875 disk.md0p1.out‚ 1003.0078125) proc.memory.availableMB‚ 1293.49609375 network.recv‚ 2.918645e+06" disk.sda2.out‚1040.69873046875 disk.sdb2.in‚3.1875 disk.sda1.in‚0 cpu‚25.205506934371012) disk./.usagePercent‚75.85285324176341% disk./.usageGB‚2712.8654441833496¯¦»ö2§"- _wandbruntime‚9072  _runtime‚9072>…þȨ" trainer/global_step‚7799 $ _timestamp‚1.788519447429645e+09 _runtime‚9072.900082795 _step‚638 ' lr-AdamW/pg1‚0.00028911908530542446 ' lr-AdamW/pg2‚0.00028911908530542446‰ª«úÍ©"Ç _runtime‚9072.900082795 _step‚638 ' lr-AdamW/pg1‚0.00028911908530542446 ' lr-AdamW/pg2‚0.00028911908530542446 trainer/global_step‚7799 $ _timestamp‚1.788519447429645e+09þ-R÷ˆ2ª"- _wandbruntime‚9072  _runtime‚9072 Zè þ'«"øO TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8866534974464 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚717670145362944 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚717670121422848 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410816 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 100107175 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚650259103775232 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚606 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚608 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚46795210 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955591168 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 146902385 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214665 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407872 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 195676702 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 195676702 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 % _timestamp‚1.7885194474308991e+09 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 _runtime‚9072.901587294 _step‚639 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1212 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9255865438208 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 356666086 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚726925986861056 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 160989382 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚650258186364416 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007241 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26088 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 100107173 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8833481028113 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9256090103808 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8866534974464 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 248605696 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411420 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 160989382 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚627079405276752 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26088 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 146901777 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 399863808 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚641391651389952 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚635912869265505 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 195650614 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8867434382336 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177047040 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚650259103775232 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 160989384 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚641391669392896 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚627079388237392 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8867434382336 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407872 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚635913783716285 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 trainer/global_step‚7799 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955591168 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26090 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚641391669392896 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8834378439533 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607'2–Q(¬"ýO _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9255865438208 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 356666086 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚726925986861056 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 160989382 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚650258186364416 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007241 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26088 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 100107173 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8833481028113 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9256090103808 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8866534974464 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 248605696 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411420 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 160989382 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚627079405276752 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26088 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 146901777 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 399863808 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚641391651389952 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚635912869265505 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 195650614 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8867434382336 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177047040 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚650259103775232 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 160989384 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚641391669392896 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚627079388237392 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8867434382336 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407872 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚635913783716285 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 trainer/global_step‚7799 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955591168 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26090 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚641391669392896 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8834378439533 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8866534974464 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚717670145362944 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚717670121422848 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410816 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 100107175 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚650259103775232 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚606 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚608 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚46795210 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955591168 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 146902385 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214665 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407872 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 195676702 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 195676702 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 % _timestamp‚1.7885194474308991e+09 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 _runtime‚9072.901587294 _step‚639 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1212 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35ÿ˜ý€2­"- _wandbruntime‚9073  _runtime‚9073¬õÅÝH®"‚N ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚641471281685504 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9257039873024 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚9073.865051499 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 195701319 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 146919877 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007241 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 356684485 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚641471299688448 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚717759440643584 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 356710601 % _timestamp‚1.7885194483938186e+09 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 356710601 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚604 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚717759464583680 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8834589689431 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26114 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚641471281685504 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 399863808 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9257260331008 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955591168 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899421184 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1187 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 N>DeviceStatsMonitor.on_t)Ì–€À rain_batch_end/requested_bytes.all.peak‚ 13510214665 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8868547198976 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚46800930 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450988 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 161009280 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚641471299688448 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177047040 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 146920481 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚46800328 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚635991958020951 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 195701319 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 195675205 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 trainer/global_step‚7799 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚650339846887424 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 100119551 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚627157368331520 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917424128 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 161009280 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 100119549 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚602 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 195675205 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177047040–8®, '¯"‡N XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8834589689431 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26114 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚641471281685504 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 399863808 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9257260331008 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955591168 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899421184 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1187 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8868547198976 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚46800930 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450988 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 161009280 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚641471299688448 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177047040 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 146920481 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚46800328 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚635991958020951 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 195701319 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 195675205 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 trainer/global_step‚7799 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚650339846887424 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 100119551 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚627157368331520 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917424128 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 161009280 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 100119549 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚602 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 195675205 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177047040 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚641471281685504 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9257039873024 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚9073.865051499 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 195701319 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 146919877 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007241 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 356684485 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚641471299688448 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚717759440643584 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 356710601 % _timestamp‚1.7885194483938186e+09 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 356710601 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚604 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚717759464583680 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0€ù°f2°"- _wandbruntime‚9073  _runtime‚9073¬Û÷\±"Ö 1loss/SRqqt1/detection_loss‚1.5639155547497636 % _timestamp‚1.7885194483959014e+09 2loss/SRqqt1/assignment_loss‚1.3582091358632356 2loss/SRqqt2/assignment_loss‚0.6794859592669776 1loss/FBt2/assignment_loss‚0.11239448449336344 0loss/SRqqt2/detection_loss‚0.956853068448485 /loss/FRt4/detection_loss‚3.1793570363003276 /loss/FRt2/assignment_loss‚1.110337014925682 2loss/SRqqt3/assignment_loss‚1.3263162652797558 1loss/FBt1/assignment_loss‚0.36187778633652506 0loss/FBt4/assignment_loss‚0.5303973569501371 /loss/FRt2/detection_loss‚1.4754334019528608 /loss/FRt3/detection_loss‚2.5693052428406338 1loss/SRqqt4/detection_loss‚2.7892109966278076 .loss/FBt3/detection_loss‚2.215588172276815 trainer/global_step‚7799 _step‚641 0loss/FRt4/assignment_loss‚2.8196354275164395 2loss/SRqqt4/assignment_loss‚1.5786082054383588 .loss/FRt1/detection_loss‚2.368012078899868 /loss/FBt4/detection_loss‚2.2509045153856277 0loss/FRt1/assignment_loss‚2.6114276336586992 .loss/FBt1/detection_loss‚2.715282670089177 /loss/FBt2/detection_loss‚0.7463509817345356 _runtime‚9073.866021985 1loss/FBt3/assignment_loss‚0.22244289881685594 1loss/SRqqt3/detection_loss‚1.6670559410955392 $loss/total_loss‚40.5154284425378 epoch‚6 0loss/FRt3/assignment_loss‚3.3070266135903292ߥæÐa²"Û 1loss/FBt1/assignment_loss‚0.36187778633652506 0loss/FBt4/assignment_loss‚0.5303973569501371 /loss/FRt2/detection_loss‚1.4754334019528608 /loss/FRt3/detection_loss‚2.5693052428406338 1loss/SRqqt4/detection_loss‚2.7892109966278076 .loss/FBt3/detection_loss‚2.215588172276815 trainer/global_step‚7799 _step‚641 0loss/FRt4/assignment_loss‚2.8196354275164395 2loss/SRqqt4/assignment_loss‚1.5786082054383588 .loss/FRt1/detection_loss‚2.368012078899868 /loss/FBt4/detection_loss‚2.2509045153856277 0loss/FRt1/assignment_loss‚2.6114276336586992 .loss/FBt1/detection_loss‚2.715282670089177 /loss/FBt2/detection_loss‚0.7463509817345356 _runtime‚9073.866021985 1loss/FBt3/assignment_loss‚0.22244289881685594 1loss/SRqqt3/detection_loss‚1.6670559410955392 $loss/total_loss‚40.5154284425378 epoch‚6 0loss/FRt3/assignment_loss‚3.3070266135903292 1loss/SRqqt1/detection_loss‚1.5639155547497636 % _timestamp‚1.7885194483959014e+09 2loss/SRqqt1/assignment_loss‚1.3582091358632356 2loss/SRqqt2/assignment_loss‚0.6794859592669776 1loss/FBt2/assignment_loss‚0.11239448449336344 0loss/SRqqt2/detection_loss‚0.956853068448485 /loss/FRt4/detection_loss‚3.1793570363003276 /loss/FRt2/assignment_loss‚1.110337014925682 2loss/SRqqt3/assignment_loss‚1.3263162652797558Pµñ’³"‚(:‡ ™ÈêԠЖÿ gpu.0.gpu‚25.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚169.61' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.4025 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0KÇÊót´"‚(:é ™ÈêÔ¬œáÿ disk.nvme0n1.in‚ 618.11328125 disk.sdb.in‚3.1875! disk.sda.out‚1042.93896484375 network.sent‚ 2.0003461e+07 disk.sdb1.out‚0 disk.md0.in‚ 9.26953125 cpu‚25.20522097814335) disk./.usagePercent‚75.85297888787026 disk.sda1.out‚0 disk.nvme1n1.in‚ 498.80078125 disk.md0.out‚ 1005.18359375" disk.sdb2.out‚1042.93896484375! disk.sdb.out‚1042.93896484375 disk.rbd0.in‚0 disk.rbd2.out‚ 2.466796875 disk.sda1.in‚0 disk.sda.in‚ 6.08203125 network.recv‚ 2.922158e+06 disk.sdb2.in‚3.1875 disk.dm-0.in‚ 1116.9140625# disk.nvme0n1.out‚50615.81640625 disk.nvme1n1.out‚ 49462.0625# proc.memory.rssMB‚ 3000.48046875) proc.memory.percent‚18.31347942352295 disk.md0p1.in‚ 9.26953125 disk.sda2.in‚ 6.08203125" disk.sda2.out‚1042.93896484375 disk.sdb1.in‚0 disk.rbd2.in‚0$ memory_percent‚92.10681915283203 disk.md0p1.out‚ 1005.18359375 disk.rbd0.out‚ 2.466796875& proc.memory.availableMB‚ 1293.21875 proc.cpu.threads‚17% disk./.usageGB‚2712.8699378967285! disk.dm-0.out‚100077.87890625Q<@电"‚(:‰ ¨ÈêÔ¼Üáÿ gpu.0.gpu‚53.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚168.051' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.01275 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÉË m¶"‚(:â ¨ÈêÔÚŸŒ€ network.sent‚ 2.0113383e+07% disk./.usageGB‚2712.8699073791504 disk.rbd0.out‚ 2.466796875 disk.rbd2.in‚0 disk.rbd2.out‚ 2.466796875 disk.sdb2.in‚3.1875) proc.memory.percent‚18.31347942352295 cpu‚25.20493632356786 disk.md0.in‚ 9.26953125 disk.nvme1n1.in‚ 498.80078125 proc.cpu.threads‚17 disk.sdb1.out‚0 disk.sda1.in‚0 network.recv‚ 2.92774e+06 disk.rbd0.in‚0 disk.dm-0.in‚ 1116.9140625 disk.sdb.out‚1044.435546875 disk.sda.in‚ 6.08203125 disk.sda1.out‚0$ memory_percent‚92.10524559020996# proc.memory.rssMB‚ 3000.48046875 disk.md0.out‚ 1006.62109375' disk./.usagePercent‚75.852978034586 disk.sda2.in‚ 6.08203125# disk.nvme1n1.out‚49462.57421875 disk.sdb1.in‚0! disk.dm-0.out‚100150.46484375 disk.md0p1.out‚ 1006.62109375 disk.nvme0n1.in‚ 618.11328125 disk.sdb2.out‚1044.435546875 disk.sdb.in‚3.1875 disk.sda2.out‚1044.435546875 disk.md0p1.in‚ 9.26953125! disk.nvme0n1.out‚ 50687.890625 disk.sda.out‚1044.435546875( proc.memory.availableMB‚ 1293.4765625Ü6«•‘·"‚(:† ·ÈêÔâ”êý gpu.0.gpu‚56.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚37.0 gpu.0.powerWatts‚114.74' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.685 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0„:t¸"‚(:é ·ÈêÔÚÿïþ# disk.nvme1n1.out‚49462.69921875 disk.dm-0.in‚ 1116.9140625 disk.sda2.in‚ 6.08203125 disk.md0.out‚ 1008.28125! disk.dm-0.out‚100150.59765625 disk.md0p1.in‚ 9.26953125 disk.sdb.in‚3.1875 network.recv‚ 2.931533e+06) disk./.usagePercent‚75.85297963449398% disk./.usageGB‚2712.8699645996094 disk.rbd0.in‚0 disk.sda1.out‚0 disk.nvme0n1.in‚ 618.11328125! disk.sdb.out‚1046.15283203125 disk.rbd2.in‚0" disk.sdb2.out‚1046.15283203125 disk.sdb1.out‚0 disk.md0.in‚ 9.26953125 disk.sdb2.in‚3.1875 disk.sda1.in‚0$ memory_percent‚92.10686683654785# proc.memory.rssMB‚ 3000.48046875 cpu‚25.204633929699824 proc.cpu.threads‚17 disk.rbd0.out‚ 2.466796875 disk.nvme1n1.in‚ 498.80078125 disk.rbd2.out‚ 2.466796875 network.sent‚ 2.0116984e+07 disk.md0p1.out‚ 1008.28125" disk.nvme0n1.out‚ 50687.8984375" disk.sda2.out‚1046.15283203125 disk.sdb1.in‚0( proc.memory.availableMB‚ 1293.2109375) proc.memory.percent‚18.31347942352295 disk.sda.in‚ 6.08203125! disk.sda.out‚1046.15283203125­ Z”¹"‚(:‰ ÆÈêÔ¤ùèý gpu.0.gpu‚40.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚170.891' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.72275 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0jMÀ8qº"‚(:æ ÆÈêÔôÒòþ disk.sdb1.in‚0! disk.dm-0.out‚100335.48046875 disk.nvme1n1.in‚ 498.80078125 disk.sdb2.in‚3.1875 disk.dm-0.in‚ 1116.9140625 disk.nvme0n1.out‚ 50872.5625 proc.cpu.threads‚17 disk.sdb1.out‚0 network.sent‚ 2.0123315e+07 disk.sda1.in‚0 disk.md0p1.out‚ 1010.30078125 disk.nvme0n1.in‚ 618.11328125 disk.sdb.out‚1048.2431640625' proc.memory.availableMB‚ 1293.203125# proc.memory.rssMB‚ 3000.48046875) disk./.usagePercent‚75.84166572520971 disk.md0.in‚ 9.26953125 disk.md0.out‚ 1010.30078125 disk.sda1.out‚0) proc.memory.percent‚18.31347942352295 network.recv‚ 2.936316e+06# disk.nvme1n1.out‚49462.91796875 disk.rbd2.out‚ 2.466796875 disk.md0p1.in‚ 9.26953125 disk.sda.out‚1048.2431640625 disk.sda2.in‚ 6.08203125 disk.rbd2.in‚0 disk.sdb.in‚3.1875 disk.sda.in‚ 6.08203125$ memory_percent‚92.10691452026367 cpu‚25.20438019344888! disk.sda2.out‚1048.2431640625! disk.sdb2.out‚1048.2431640625% disk./.usageGB‚2712.4653244018555 disk.rbd0.in‚0 disk.rbd0.out‚ 2.4667968754äî2»"- _wandbruntime‚9124  _runtime‚9124—êÇǼ"Á _runtime‚9124.540369597 _step‚642 & lr-AdamW/pg1‚0.0002832807502230089 & lr-AdamW/pg2‚0.0002832807502230089 trainer/global_step‚7849 % _timestamp‚1.7885194990699234e+09§Ÿ¨S̽"Æ & lr-AdamW/pg2‚0.0002832807502230089 trainer/global_step‚7849 % _timestamp‚1.7885194990699234e+09 _runtime‚9124.540369597 _step‚642 & lr-AdamW/pg1‚0.0002832807502230089‚¨}EË2¾"- _wandbruntime‚9124  _runtime‚9124*v ξ&¿"øO QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450780 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚639867320629013 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8888927685221 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 161984282 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9314437410304 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007241 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 196883516 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917410816 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8923088962560 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 233925632 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8922189554688 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410816 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚47081979 _step‚643 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚47082556 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 196909604 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8923088962560 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407872 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 209985536 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚645373184170496 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚722136082459648 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚630978392943792 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 161984284 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955591168 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚577 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 399863808 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9314227424768 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8922189554688 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214665 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955591168 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚654296273133056 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26088 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 358867798 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 161984282 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 358893888 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 147807952 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚645373166167552 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 196909604 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚731450543810048 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚722136106399744 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚630978409983152 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚654296273133056 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚654295355722240 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚654295355722240 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 161984284 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚645373166167552 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26090 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚579 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7885194990711296e+09 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26088 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _runtime‚9124.541775224 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1212 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1187 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_start/allocation.laï©@rge_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 147808531 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411420 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8889825096641 trainer/global_step‚7849srG¡(À"ýO JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 196883516 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917410816 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8923088962560 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 233925632 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8922189554688 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410816 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚47081979 _step‚643 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚47082556 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 196909604 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8923088962560 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407872 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 209985536 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚645373184170496 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚722136082459648 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚630978392943792 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 161984284 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955591168 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚577 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 399863808 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9314227424768 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8922189554688 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214665 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955591168 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚654296273133056 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26088 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 358867798 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 161984282 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 358893888 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 147807952 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚645373166167552 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 196909604 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚731450543810048 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚722136106399744 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚630978409983152 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚654296273133056 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚654295355722240 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚654295355722240 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 161984284 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚645373166167552 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26090 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚579 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7885194990711296e+09 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26088 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _runtime‚9124.541775224 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1212 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1187 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 147808531 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411420 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8889825096641 trainer/global_step‚7849 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450780 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚639867320629013 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8888927685221 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 161984282 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9314437410304 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007241ƒlÃu2Á"- _wandbruntime‚9125  _runtime‚9125s®Q•'Â"N QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177047040 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 _runtime‚9125.538511488 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 196908107 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9315413562880 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899421184 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 399863808 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26114 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚591 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚639947323835447 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚631056390077280 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8890933758167 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚654377016245248 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 100738351 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 231815168 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚722225425620480 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8923302358016 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411628 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 162004180 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 trainer/global_step‚7849 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177047040 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚654376098821120 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31686 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 358912287 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 196934221 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚639946409384459 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚645452796463104 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 196934221 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 147826662 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚593 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 _step‚644 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 358938403 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917424128 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 100738349 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚654377016245248 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚47087720 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 162004180 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 162004182 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚645452814466048 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8924201779200 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1187 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 162004182 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚731540815243264 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚722225401680384 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8890036346539 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899421184 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚47088311 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450988 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 207875072 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26114 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚645452796463104 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 196908107 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 358938403 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚731541047058432 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚631056373037920 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚645452814466048 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 147826069 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9315621437952 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4 ( 'Ã"†N UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8890036346539 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899421184 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚47088311 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450988 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 207875072 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26114 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚645452796463104 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 196908107 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 358938403 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚731541047058432 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚631056373037920 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚645452814466048 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 147826069 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9315621437952 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177047040 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 _runtime‚9125.538511488 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 196908107 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9315413562880 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899421184 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 399863808 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26114 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚591 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚639947323835447 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚631056390077280 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8890933758167 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚654377016245248 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 100738351 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 231815168 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚722225425620480 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8923302358016 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411628 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 162004180 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 trainer/global_step‚7849 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177047040 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚654376098821120 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31686 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 358912287 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 196934221 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚639946409384459 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚645452796463104 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 196934221 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 147826662 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚593 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 _step‚644 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 358938403 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917424128 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 100738349 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚654377016245248 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚47087720 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 162004180 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 162004182 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚645452814466048 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8924201779200 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1187 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 162004182 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚731540815243264 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚722225401680384„ðZÞÀ2Ä"- _wandbruntime‚9125  _runtime‚91254œ']Å"× .loss/FBt1/detection_loss‚2.124583141370253 $loss/total_loss‚38.0404349164767 _step‚645 /loss/FRt1/detection_loss‚1.9633247305334427 /loss/FRt3/detection_loss‚2.0150810528445886 trainer/global_step‚7849 epoch‚6 0loss/FRt1/assignment_loss‚2.2928984976829367 0loss/FRt2/assignment_loss‚1.4459598525877921 1loss/SRqqt3/assignment_loss‚1.328387438133359 /loss/FBt2/detection_loss‚0.8223024977105005 /loss/FRt4/assignment_loss‚3.561963110058396 1loss/SRqqt3/detection_loss‚1.5368925588826339 1loss/SRqqt4/detection_loss‚2.0797830279916525 .loss/FBt4/detection_loss‚3.092380426824093 % _timestamp‚1.7885195000693717e+09 /loss/FRt2/detection_loss‚1.5188039718496222 0loss/FRt3/assignment_loss‚2.2060441648637927 1loss/FBt2/assignment_loss‚0.09617274508440476 3loss/FBt4/assignment_loss‚0.0012092882641354663 .loss/FRt4/detection_loss‚3.167821274863349 1loss/SRqqt2/detection_loss‚0.8174231219519343 /loss/FBt3/detection_loss‚2.3611221313476562 _runtime‚9125.539515273 2loss/SRqqt2/assignment_loss‚0.8924779956060875 2loss/SRqqt4/assignment_loss‚1.6475309415254742 1loss/FBt3/assignment_loss‚0.25189659477910026 0loss/SRqqt1/detection_loss‚1.537438331240857 1loss/SRqqt1/assignment_loss‚1.199307758981983 1loss/FBt1/assignment_loss‚0.07963026149865403/k}Š±Æ"Ü .loss/FBt1/detection_loss‚2.124583141370253 $loss/total_loss‚38.0404349164767 _step‚645 /loss/FRt1/detection_loss‚1.9633247305334427 /loss/FRt3/detection_loss‚2.0150810528445886 trainer/global_step‚7849 epoch‚6 0loss/FRt1/assignment_loss‚2.2928984976829367 0loss/FRt2/assignment_loss‚1.4459598525877921 1loss/SRqqt3/assignment_loss‚1.328387438133359 /loss/FBt2/detection_loss‚0.8223024977105005 /loss/FRt4/assignment_loss‚3.561963110058396 1loss/SRqqt3/detection_loss‚1.5368925588826339 1loss/SRqqt4/detection_loss‚2.0797830279916525 .loss/FBt4/detection_loss‚3.092380426824093 % _timestamp‚1.7885195000693717e+09 /kùð˱loss/FRt2/detection_loss‚1.5188039718496222 0loss/FRt3/assignment_loss‚2.2060441648637927 1loss/FBt2/assignment_loss‚0.09617274508440476 3loss/FBt4/assignment_loss‚0.0012092882641354663 .loss/FRt4/detection_loss‚3.167821274863349 1loss/SRqqt2/detection_loss‚0.8174231219519343 /loss/FBt3/detection_loss‚2.3611221313476562 _runtime‚9125.539515273 2loss/SRqqt2/assignment_loss‚0.8924779956060875 2loss/SRqqt4/assignment_loss‚1.6475309415254742 1loss/FBt3/assignment_loss‚0.25189659477910026 0loss/SRqqt1/detection_loss‚1.537438331240857 1loss/SRqqt1/assignment_loss‚1.199307758981983 1loss/FBt1/assignment_loss‚0.07963026149865403…Ž!t”Ç"‚(:‰ ÕÈêÔ…÷ € gpu.0.gpu‚50.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚168.383' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.09575 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0X½fÈ"‚(:Û ÕÈêÔçšó€ disk.sda2.in‚ 6.08203125 disk.rbd0.in‚0 disk.sdb2.out‚1049.943359375 disk.sdb.in‚3.1875 cpu‚25.20408855057491 disk.rbd2.out‚ 2.466796875 disk.md0p1.out‚ 1011.9375( proc.memory.availableMB‚ 1293.8828125 disk.md0.in‚ 9.26953125! disk.dm-0.out‚100335.67578125) disk./.usagePercent‚75.85219237310639 disk.sda.out‚1049.943359375 disk.sdb2.in‚3.1875 disk.dm-0.in‚ 1116.9140625 disk.md0p1.in‚ 9.26953125# disk.nvme0n1.out‚50872.62890625 disk.sdb.out‚1049.943359375# proc.memory.rssMB‚ 3000.48046875$ disk./.usageGB‚2712.841808319092 disk.rbd0.out‚ 2.466796875 disk.nvme1n1.in‚ 498.80078125 disk.sdb1.out‚0 disk.sda1.in‚0 disk.nvme0n1.in‚ 618.11328125 proc.cpu.threads‚17 network.sent‚ 2.0140616e+07 network.recv‚ 2.938913e+06 disk.sda2.out‚1049.943359375 disk.sda.in‚ 6.08203125! disk.nvme1n1.out‚ 49463.046875 disk.md0.out‚ 1011.9375 disk.rbd2.in‚0 disk.sda1.out‚0# memory_percent‚92.1027660369873) proc.memory.percent‚18.31347942352295 disk.sdb1.in‚0¸“”É"‚(:‰ äÈêÔéë›þ gpu.0.gpu‚47.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚37.0 gpu.0.powerWatts‚115.025' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.75625 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0VWÕtÊ"‚(:é äÈêÔ“êÝÿ network.sent‚ 2.023553e+07 disk.sda.in‚ 6.08203125 disk.rbd0.out‚ 2.466796875 disk.nvme1n1.in‚ 498.80078125 disk.sdb1.out‚0! disk.sdb2.out‚1051.4677734375 disk.sdb.out‚1051.4677734375# proc.memory.rssMB‚ 3000.48046875 disk.nvme0n1.in‚ 618.11328125% disk./.usageGB‚2712.7775115966797" disk.nvme1n1.out‚ 49464.2578125 disk.md0.in‚ 9.26953125 disk.rbd2.out‚ 2.466796875 disk.dm-0.in‚ 1116.9140625 disk.sda.out‚1051.4677734375 disk.sda2.in‚ 6.08203125 disk.rbd0.in‚0 disk.sdb1.in‚0 disk.md0.out‚ 1013.40234375 disk.sda1.in‚0 disk.md0p1.out‚ 1013.40234375) proc.memory.percent‚18.31347942352295 disk.sdb2.in‚3.1875 disk.sdb.in‚3.1875! disk.dm-0.out‚100659.75390625 cpu‚25.20381618913486# disk.nvme0n1.out‚51195.49609375) disk./.usagePercent‚75.85039460983748 disk.rbd2.in‚0 disk.sda1.out‚0$ memory_percent‚92.10431575775146 network.recv‚ 2.94426e+06) proc.memory.availableMB‚ 1293.62890625 proc.cpu.threads‚17! disk.sda2.out‚1051.4677734375 disk.md0p1.in‚ 9.26953125Šv|¾”Ë"‚(:‰ óÈêÔ¡þ‡þ gpu.0.gpu‚54.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚150.391' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚37.59775 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0䫤ùgÌ"‚(:Ü óÈêÔƶËÿ! disk.sda.out‚1052.90185546875 disk.md0.in‚ 9.26953125 disk.sdb1.in‚0 disk.rbd2.out‚ 2.466796875# proc.memory.rssMB‚ 3000.48046875 disk.nvme1n1.in‚ 498.80078125 disk.rbd2.in‚0) disk./.usagePercent‚75.85290251892926 disk.sdb2.in‚3.1875 proc.cpu.threads‚17 disk.nvme0n1.out‚51195.5 disk.rbd0.in‚0$ memory_percent‚92.10431575775146) proc.memory.availableMB‚ 1293.62890625 network.recv‚ 2.9482e+06 disk.sda1.in‚0 disk.dm-0.in‚ 1116.9140625! disk.sdb.out‚1052.90185546875 disk.sda2.in‚ 6.08203125 disk.md0.out‚ 1014.78125) proc.memory.percent‚18.31347942352295 disk.dm-0.out‚ 100747.96875 disk.md0p1.out‚ 1014.78125 disk.sda.in‚ 6.08203125 cpu‚25.203538785812242 disk.sdb1.out‚0 network.sent‚ 2.0241334e+07% disk./.usageGB‚2712.8672065734863 disk.md0p1.in‚ 9.26953125 disk.sdb.in‚3.1875" disk.sda2.out‚1052.90185546875 disk.rbd0.out‚ 2.466796875 disk.nvme1n1.out‚ 49552.46875" disk.sdb2.out‚1052.90185546875 disk.sda1.out‚0 disk.nvme0n1.in‚ 618.11328125c9"‹{Í"‚(:ð ‚ÉêÔÄæ”ÿ disk.dm-0.out‚100965.0390625 disk.rbd0.out‚ 2.466796875" disk.sdb2.out‚1054.81005859375 disk.sda.in‚ 6.08203125 disk.rbd0.in‚0 disk.sdb1.out‚0% disk./.usageGB‚2713.3700828552246# disk.nvme0n1.out‚51405.50390625# disk.nvme1n1.out‚49559.53515625) proc.memory.availableMB‚ 1293.70703125 network.recv‚ 2.953245e+06 disk.md0.in‚ 9.26953125 disk.sdb.in‚3.1875 disk.sda2.in‚ 6.08203125$ memory_percent‚92.10383892059326 network.sent‚ 2.0247769e+07 disk.rbd2.in‚0 disk.sdb2.in‚3.1875 disk.sda1.in‚0 disk.sda1.out‚0 disk.md0p1.out‚ 1016.62890625 disk.dm-0.in‚ 1116.9140625! disk.sdb.out‚1054.81005859375 disk.sdb1.in‚0# proc.memory.rssMB‚ 3000.48046875) proc.memory.percent‚18.31347942352295 proc.cpu.threads‚17 disk.md0.out‚ 1016.62890625! disk.sda.out‚1054.81005859375" disk.sda2.out‚1054.81005859375 disk.nvme1n1.in‚ 498.80078125) disk./.usagePercent‚75.86696315023673 disk.rbd2.out‚ 2.466796875 disk.md0p1.in‚ 9.26953125 disk.nvme0n1.in‚ 618.11328125 cpu‚25.203263834407107ÀžÉb”Î"‚(:‰ ‚ÉêÔ´Ž°ÿ gpu.0.gpu‚54.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚171.501' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.87525 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‹„™K2Ï"- _wandbruntime‚9180  _runtime‚9180þ¼UÉÐ"à trainer/global_step‚7899 % _timestamp‚1.7885195549323428e+09 _runtime‚9180.402814302 _step‚646 ' lr-AdamW/pg1‚0.00027747275161662437 ' lr-AdamW/pg2‚0.00027747275161662437¶j ÎÑ"È ' lr-AdamW/pg2‚0.00027747275161662437 trainer/global_step‚7899 % _timestamp‚1.7885195549323428e+09 _runtime‚9180.402814302 _step‚646 ' lr-AdamW/pg1‚0.00027747275161662437†t¬j­2Ò"- _wandbruntime‚9180  _runtime‚9180Ì*ˆmþ'Ó"øO WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚735974886025216 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚735974631128064 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚658333442490880 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407872 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 162979184 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚634877414689552 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955591168 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8977844134912 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 198116418 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 361095600 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 148714274 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 254897152 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9372818588672 trainer/global_step‚7899 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚649354680945152 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚658333442490880 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450780 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 162979184 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1187 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚635 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 148713637 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26088 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8944374342329 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚649354698948096 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407872 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8977844134912 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26090 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 361121690 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007241 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚649354680945152 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 230957056 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 162979182 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 101344773 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚658332525080064 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 399863808 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚726602067436544 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚643821771992521 % _timestamp‚1.7885195549330928e+09 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8978743542784 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 198142506 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26088 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 101344775 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚634877397650192 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8945271753749 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955591168 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚47369499 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚47368864 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 198116418 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410816 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚658332525080064 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177047040 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚637 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚726602043496448 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8978743542784 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 198142506 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917410816 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚649354698948096 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9372587631616 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411420 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171¥õ(Ô"ýO WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8977844134912 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26090 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 361121690 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007241 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚649354680945152 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 230957056 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 162979182 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 101344773 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚658332525080064 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 399863808 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚726602067436544 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚643821771992521 % _timestamp‚1.7885195549330928e+09 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8978743542784 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 198142506 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26088 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 101344775 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚634877397650192 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8945271753749 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955591168 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚47369499 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚47368864 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 198116418 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410816 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚658332525080064 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177047040 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚637 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚726602043496448 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8978743542784 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 198142506 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917410816 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚649354698948096 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9372587631616 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411420 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚735974886025216 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚735974631128064 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚658333442490880 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407872 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 162979184 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚634877414689552 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955591168 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8977844134912 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 198116418 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 361095600 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 148714274 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 254897152 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9372818588672 trainer/global_step‚7899 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚649354680945152 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚658333442490880 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450780 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 162979184 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1187 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚635 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 148713637 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26088 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8944374342329 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚649354698948096 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407872‡_f`¥2Õ"- _wandbruntime‚9181  _runtime‚9181š¯žÖ"‚N WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚649434311240704 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚643901775198955 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 162999082 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411628 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899421184 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚640 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9373748676608 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚643900860747967 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚634955377744320 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8978956938240 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚649434329243648 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26114 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 162999082 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 162999080 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 148732376 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 361140089 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 101357151 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 399863808 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 361140089 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 361166205 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 226749440 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450988 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007241 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 198141009 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917424128 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31686 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚736065111393792 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8945483003647 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 198141009 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 250689536 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 148731734 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚47375225 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 198167123 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚658413268178944 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8979856359424 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚726691362717184 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚634955394783680 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26114 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚658414185603072 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 361166205 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955591168 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚649434329243648 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚658414185603072 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 198167123 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9373975426048 _runtime‚9181.391177857 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 101357149 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177047040 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8946380415275žu´ '×"‡N D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 361140089 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 361166205 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 226749440 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450988 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007241 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 198141009 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917424128 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31686 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚736065111393792 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8945483003647 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 198141009 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 250689536 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 148731734 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚47375225 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 198167123 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚658413268178944 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8979856359424 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚726691362717184 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚634955394783680 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26114 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚658414185603072 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 361166205 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955591168 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚649434329243648 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚658414185603072 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 198167123 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9373975426048 _runtime‚9181.391177857 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 101357149 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177047040 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8946380415275 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚649434311240704 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚643901775198955 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 162999082 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411628 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899421184 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚640 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9373748676608 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚643900860747967 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚634955377744320 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8978956938240 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚649434329243648 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26114 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 162999082 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 162999080 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 148732376 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 361140089 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 101357151 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 399863808ˆÝ’,Û2Ø"- _wandbruntime‚9181  _runtime‚9181Ž2]B^Ù"Ø 1loss/SRqqt1/detection_loss‚1.3257546276418548 trainer/global_step‚7899 0loss/SRqqt3/detection_loss‚1.579150512852535 2loss/SRqqt2/assignment_loss‚0.8679063964536358 /loss/FRt1/detection_loss‚2.0065568209660873 .loss/FBt3/detection_loss‚2.954738905032476 /loss/FBt4/detection_loss‚2.8343944450219474 2loss/FBt2/assignment_loss‚0.047706794876036664 1loss/FBt3/assignment_loss‚0.24347952820001714 /loss/FRt4/detection_loss‚3.0563132667541506 1loss/SRqqt4/detection_loss‚2.2789464108645916 /loss/FBt2/detection_loss‚0.6582239168169706 /loss/FRt4/assignment_loss‚2.812326068878174 2loss/SRqqt1/assignment_loss‚1.3215406933855829 0loss/SRqqt2/detection_loss‚1.015603702176701 /loss/FBt1/detection_loss‚1.7358265885939965 &loss/total_loss‚38.783667958111685 _runtime‚9181.392770954 /loss/FRt3/assignment_loss‚2.586605166516653 % _timestamp‚1.7885195559226313e+09 0loss/FRt2/assignment_loss‚1.8696480955717698 1loss/FBt1/assignment_loss‚0.12749062416744786 0loss/FBt4/assignment_loss‚0.6095016147113105 epoch‚6 0loss/FRt1/assignment_loss‚2.1118633574909635 2loss/SRqqt3/assignment_loss‚1.2749727742404353 1loss/SRqqt4/assignment_loss‚1.379923991393298 /loss/FRt2/detection_loss‚1.6046544097469382 /loss/FRt3/detection_loss‚2.4805392457581146 _step‚649&úncÚ"Ý 1loss/SRqqt1/detection_loss‚1.3257546276418548 trainer/global_step‚7899 0loss/SRqqt3/detection_loss‚1.579150512852535 2loss/SRqqt2/assignment_loss‚0.8679063964536358 /loss/FRt1/detection_loss‚2.0065568209660873 .loss/FBt3/detection_loss‚2.954738905032476 /loss/FBt4/detection_loss‚2.8343944450219474 2loss/FBt2/assignment_loss‚0.047706794876036664 1loss/FBt3/assignment_loss‚0.24347952820001714 /loss/FRt4/detection_loss‚3.0563132667541506 1loss/SRqqt4/detection_loss‚2.2789464108645916 /loss/FBt2/detection_loss‚0.6582239168169706 /loss/FRt4/assignment_loss‚2.812326068878174 2loss/SRqqt1/assignment_loss‚1.3215406933855829 0loss/SRqqt2/detection_loss‚1.015603702176701 /loss/FBt1/detection_loss‚1.7358265885939965 &loss/total_loss‚38.783667958111685 _runtime‚9181.392770954 /loss/FRt3/assignment_loss‚2.586605166516653 % _timestamp‚1.7885195559226313e+09 0loss/FRt2/assignment_loss‚1.8696480955717698 1loss/FBt1/assignment_loss‚0.12749062416744786 0loss/FBt4/assignment_loss‚0.6095016147113105 epoch‚6 0loss/FRt1/assignment_loss‚2.1118633574909635 2loss/SRqqt3/assignment_loss‚1.2749727742404353 1loss/SRqqt4/assignment_loss‚1.379923991393298 /loss/FRt2/detection_loss‚1.6046544097469382 /loss/FRt3/detection_loss‚2.4805392457581146 _step‚649‰œ¾K¼’Û"‚(:‡ ‘ÉêÔÂ×»þ gpu.0.gpu‚55.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚37.0 gpu.0.powerWatts‚121.536' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚30.384 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0²«ÔÂxÜ"‚(:í ‘ÉêÔ­¢üþ disk.sdb2.in‚3.1875 disk.nvme0n1.in‚ 618.11328125 disk.sda2.in‚ 6.08203125" disk.sda2.out‚1056.39794921875$ memory_percent‚92.10255146026611# proc.memory.rssMB‚ 3000.48046875 network.recv‚ 2.956676e+06 disk.md0p1.out‚ 1018.16796875) disk./.usagePercent‚75.87046556213002 disk.rbd2.in‚0 disk.sda.in‚ 6.08203125! disk.sda.out‚1056.39794921875) proc.memory.percent‚18.31347942352295" disk.sdb2.out‚1056.39794921875 disk.nvme1n1.in‚ 498.80078125 disk.sdb1.in‚0 disk.sda1.in‚0# disk.nvme0n1.out‚51510.44140625 cpu‚25.202988443472453 proc.cpu.threads‚17$ disk./.usageGB‚2713.495346069336# disk.nvme1n1.out‚49595.65234375 disk.sdb1.out‚0 disk.md0.out‚ 1018.16796875) proc.memory.availableMB‚ 1293.91796875 disk.rbd2.out‚ 2.466796875 disk.sda1.out‚0! disk.sdb.out‚1056.39794921875 disk.rbd0.in‚0 network.sent‚ 2.0253027e+07 disk.dm-0.in‚ 1116.9140625 disk.dm-0.out‚ 101106.09375 disk.md0p1.in‚ 9.26953125 disk.sdb.in‚3.1875 disk.rbd0.out‚ 2.466796875 disk.md0.in‚ 9.26953125±Ì¿cžÝ"‚(:“  ÉêÔجìý gpu.0.gpu‚23.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚118.474' gpu.0.enforcedPowerLimitWatts‚400.0) gpu.0.powerPercent‚29.618500000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‚<vÞ"‚(:ë  ÉêÔŽ¿Ïÿ disk.rbd2.out‚ 2.466796875) disk./.usagePercent‚75.87396594747318# disk.nvme0n1.out‚51615.48046875! disk.sda.out‚1057.83544921875 disk.sdb2.in‚3.1875) proc.memory.percent‚18.31347942352295 disk.rbd2.in‚0 network.recv‚ 2.962057e+06 disk.sda1.out‚0 disk.md0p1.in‚ 9.26953125 disk.sdb.in‚3.1875" disk.nvme1n1.out‚ 49632.2890625 disk.nvme0n1.in‚ 618.11328125 disk.rbd0.in‚0 disk.md0.in‚ 9.26953125# memory_percent‚92.1010971069336 disk.sda.in‚ 6.08203125# proc.memory.rssMB‚ 3000.48046875 proc.cpu.threads‚17 disk.md0.out‚ 1019.55078125$ disk./.usageGB‚2713.620536804199 disk.dm-0.in‚ 1116.9140625! disk.dm-0.out‚101247.76953125 disk.sda2.in‚ 6.08203125 disk.sdb1.in‚0& proc.memory.availableMB‚ 1294.15625 cpu‚25.202736356291073 disk.md0p1.out‚ 1019.55078125 disk.rbd0.out‚ 2.466796875 disk.nvme1n1.in‚ 498.80078125 disk.sda1.in‚0 disk.sdb1.out‚0" disk.sdb2.out‚1057.83544921875 network.sent‚ 2.0360939e+07! disk.sdb.out‚1057.83544921875" disk.sda2.out‚1057.83544921875ZޞΔß"‚(:‰ ¯ÉêÔÄÍ„þ gpu.0.gpu‚53.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚36.0 gpu.0.powerWatts‚143.263' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚35.81575 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0/‘8®sà"‚(:è ¯ÉêÔ¦¿‰ÿ disk.sda1.out‚0" disk.nvme0n1.out‚ 51825.1328125 cpu‚25.202464698822762 disk.md0p1.in‚ 9.26953125 disk.sdb.in‚3.1875 disk.nvme1n1.in‚ 498.80078125 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.rbd2.out‚ 2.466796875 disk.sda.out‚1059.8310546875# memory_percent‚92.1010971069336 disk.rbd0.in‚0 disk.md0p1.out‚ 1021.49609375) proc.memory.percent‚18.31347942352295) disk./.usagePercent‚75.87395314820932 disk.md0.out‚ 1021.49609375 disk.rbd2.in‚0 disk.sda1.in‚0& proc.memory.availableMB‚ 1294.15625 proc.cpu.threads‚17 disk.sdb.out‚1059.8310546875 network.recv‚ 2.966553e+06# disk.nvme1n1.out‚49812.08984375! disk.sdb2.out‚1059.8310546875! disk.dm-0.out‚101637.22265625% disk./.usageGB‚2713.6200790405273! disk.sda2.out‚1059.8310546875 disk.rbd0.out‚ 2.466796875 disk.nvme0n1.in‚ 618.11328125# proc.memory.rssMB‚ 3000.48046875 disk.sda2.in‚ 6.08203125 disk.md0.in‚ 9.26953125 disk.sdb2.in‚3.1875 disk.dm-0.in‚ 1116.9140625 disk.sda.in‚ 6.08203125 network.sent‚ 2.0367213e+07¦d€2á"- _wandbruntime‚9238  _runtime‚9238fŒÌaÈâ" ' lr-AdamW/pg2‚0.00027169629986447503 trainer/global_step‚7949 $ _timestamp‚1.788519613280751e+09 _runtime‚9238.751202082 _step‚650 ' lr-AdamW/pg1‚0.00027169629986447503¶ûËÍã"Ç $ _timestamp‚1.788519613280751e+09 _runtime‚9238.751202082 _step‚650 ' lr-AdamW/pg1‚0.00027169629986447503 ' lr-AdamW/pg2‚0.00027169629986447503 trainer/global_step‚7949Š?Ï_2ä"- _wandbruntime‚9238  _runtime‚9238Îbˆ%å"øO eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚731068028473344 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚47657304 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9033498715136 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 163974084 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 399863808 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 199349320 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚647777137806809 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214665 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 199349320 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚653336195722752 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 149621497 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955591168 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 101963575 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 230957056 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9430988261376 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 199375408 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚653336213725696 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚620 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26088 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚47657922 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚731068004533248 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450780 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 trainer/global_step‚7949 _runtime‚9238.753317472 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9000718410857 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚662370611848704 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407872 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 163974082 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9431219218432 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚662369694437888 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9034398123008 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚647776223356029 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 163974084 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚653336195722752 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 199375408 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 KBDeviceStatsMonitor.on_train_batch_start/allocation.la )ícörge_pool.peak‚3677 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411420 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚740498992794624 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26088 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚47657922 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚731068004533248 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450780 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 trainer/global_step‚7949 _runtime‚9238.753317472 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9000718410857 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚662370611848704 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407872 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 163974082 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9431219218432 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚662369694437888 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9034398123008 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚647776223356029 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 163974084 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚653336195722752 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 199375408 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411420 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚740498992794624 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 101963575 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 230957056 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9430988261376 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 199375408 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚653336213725696 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚620 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31686 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 363367891 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 199400025 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚653415844021248 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚47663604 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚647856226562463 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 _runtime‚9239.836636655 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 199373911 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007241 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚653415844021248 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955591168 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899421184 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 226749440 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450988 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 trainer/global_step‚7949 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 101975949 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 149638954 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9034611518464 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚731157347694080 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1187 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 % _timestamp‚1.7885196143654492e+09 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚662450437536768 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9034611518464 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9000929660755 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 363394007 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177047040 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917424128 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 363367891 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚638854399490080 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚662451354960896 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 163993980 _step‚652 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚47663005 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 399863808 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 199400025 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚740589724067328 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9432376373248 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚662451354960896 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚653415826018304 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26114 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚601 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 250689536 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚740589473377792 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚731157323753984 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9001827072383 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 149638954 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9034611518464 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚731157347694080 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1187 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 % _timestamp‚1.7885196143654492e+09 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚662450437536768 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9034611518464 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9000929660755 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 363394007 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177047040 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917424128 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 363367891 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚638854399490080 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚662451354960896 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 163993980 _step‚652 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚47663005 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 399863808 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 199400025 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚740589724067328 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9432376373248 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚662451354960896 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚653415826018304 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26114 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚601 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 250689536 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚740589473377792 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚731157323753984 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9001827072383 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31686 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 363367891 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 199400025 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚653415844021248 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚47663604 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚647856226562463 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 _runtime‚9239.836636655 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 199373911 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007241 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚653415844021248 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955591168 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899421184 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 226749440 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450988 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677Œ»·zy2ê"- _wandbruntime‚9239  _runtime‚9239¦‹‹[ë"Õ 1loss/SRqqt4/detection_loss‚2.5789915631557334 .loss/FBt3/detection_loss‚2.586343377828598 /loss/FBt4/detection_loss‚3.1441361804803214 _runtime‚ 9239.83762001 1loss/FBt2/assignment_loss‚0.08358485631598526 .loss/FRt3/detection_loss‚2.420855313200842 1loss/FBt4/assignment_loss‚0.14639735523572503 .loss/FRt1/detection_loss‚1.791969701643878 %loss/total_loss‚38.82223023114942 epoch‚6 0loss/FRt1/assignment_loss‚2.2052432470851473 0loss/FRt3/assignment_loss‚2.6249453249302777 /loss/FRt4/assignment_loss‚3.492789552761958 1loss/FBt3/assignment_loss‚0.13706841788734891 1loss/SRqqt1/assignment_loss‚1.174146312728243 2loss/SRqqt4/assignment_loss‚1.9594344481311996 /loss/FRt4/detection_loss‚2.9845674954927883 1loss/SRqqt1/detection_loss‚1.6789214741087184 .loss/FBt1/detection_loss‚2.408187299966812 1loss/SRqqt2/detection_loss‚1.0646536260270156 $ _timestamp‚1.788519614367446e+09 _step‚653 trainer/global_step‚7949 0loss/FRt2/assignment_loss‚1.0651115732533591 2loss/SRqqt2/assignment_loss‚1.0277757150909075 1loss/FBt1/assignment_loss‚0.05903882980680919 0loss/FBt2/detection_loss‚0.41930331090856116 2loss/SRqqt3/assignment_loss‚1.0068923784098749 /loss/FRt2/detection_loss‚1.3978621011732943 0loss/SRqqt3/detection_loss‚1.364010775526022>ÅgÂûì"Ú trainer/global_step‚7949 0loss/FRt2/assignment_loss‚1.0651115732533591 2loss/SRqqt2/assignment_loss‚1.0277757150909075 1loss/FBt1/assignment_loss‚0.05903882980680919 0loss/FBt2/detection_loss‚0.41930331090856116 2loss/SRqpõá¡eqt3/assignment_loss‚1.0068923784098749 /loss/FRt2/detection_loss‚1.3978621011732943 0loss/SRqqt3/detection_loss‚1.364010775526022 1loss/SRqqt4/detection_loss‚2.5789915631557334 .loss/FBt3/detection_loss‚2.586343377828598 /loss/FBt4/detection_loss‚3.1441361804803214 _runtime‚ 9239.83762001 1loss/FBt2/assignment_loss‚0.08358485631598526 .loss/FRt3/detection_loss‚2.420855313200842 1loss/FBt4/assignment_loss‚0.14639735523572503 .loss/FRt1/detection_loss‚1.791969701643878 %loss/total_loss‚38.82223023114942 epoch‚6 0loss/FRt1/assignment_loss‚2.2052432470851473 0loss/FRt3/assignment_loss‚2.6249453249302777 /loss/FRt4/assignment_loss‚3.492789552761958 1loss/FBt3/assignment_loss‚0.13706841788734891 1loss/SRqqt1/assignment_loss‚1.174146312728243 2loss/SRqqt4/assignment_loss‚1.9594344481311996 /loss/FRt4/detection_loss‚2.9845674954927883 1loss/SRqqt1/detection_loss‚1.6789214741087184 .loss/FBt1/detection_loss‚2.408187299966812 1loss/SRqqt2/detection_loss‚1.0646536260270156 $ _timestamp‚1.788519614367446e+09 _step‚653´$»’í"‚(:‡ ¾ÉêÔí¤æþ gpu.0.gpu‚52.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚36.0 gpu.0.powerWatts‚114.408' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.602 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0VSó"_î"‚(:Ô ¾ÉêԔƓÿ disk.md0.in‚ 9.26953125 disk.rbd2.in‚0 disk.sdb2.in‚3.1875 network.recv‚ 2.971576e+06 disk.sda2.in‚ 6.08203125 network.sent‚ 2.037352e+07' disk./.usagePercent‚75.880961598454 disk.sda.in‚ 6.08203125 disk.sda.out‚ 1061.359375 disk.sda2.out‚ 1061.359375 disk.md0.out‚ 1022.97265625 disk.rbd2.out‚ 2.466796875 disk.sdb2.out‚ 1061.359375$ disk./.usageGB‚2713.870735168457 disk.sda1.in‚0 disk.sda1.out‚0 disk.nvme1n1.in‚ 498.80078125 proc.cpu.threads‚17 disk.dm-0.out‚101637.5703125 disk.rbd0.in‚0 disk.dm-0.in‚ 1116.9140625 disk.nvme0n1.in‚ 618.11328125 disk.sdb.in‚3.1875) proc.memory.percent‚18.31347942352295 disk.rbd0.out‚ 2.466796875# proc.memory.rssMB‚ 3000.48046875 cpu‚25.202190437834275 disk.sdb.out‚ 1061.359375 disk.nvme1n1.out‚ 49812.34375 disk.sdb1.in‚0$ memory_percent‚92.10398197174072) proc.memory.availableMB‚ 1293.68359375 disk.md0p1.in‚ 9.26953125 disk.md0p1.out‚ 1022.97265625" disk.nvme0n1.out‚ 51825.2265625 disk.sdb1.out‚0æq§u”ï"‚(:‰ ÍÉêÔî™Äý gpu.0.gpu‚50.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚172.395' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.09875 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Öþ¶ uð"‚(:ê ÍÉêÔÕÓØþ disk.nvme1n1.in‚ 498.80078125! disk.sda2.out‚1062.8818359375) proc.memory.percent‚18.31347942352295! disk.nvme1n1.out‚ 49856.484375 disk.rbd2.in‚0 disk.rbd2.out‚ 2.466796875 disk.sda.in‚ 6.08203125# proc.memory.rssMB‚ 3000.48046875 proc.cpu.threads‚17 disk.sdb1.in‚0 disk.sda1.out‚0# disk.nvme0n1.out‚51961.21484375 cpu‚25.201920149386996 network.recv‚ 2.976386e+06 disk.rbd0.out‚ 2.466796875! disk.sdb2.out‚1062.8818359375 disk.md0p1.out‚ 1024.44140625( proc.memory.availableMB‚ 1293.8515625 disk.md0p1.in‚ 9.26953125 disk.sdb.in‚3.1875 disk.sdb.out‚1062.8818359375 disk.sda.out‚1062.8818359375) disk./.usagePercent‚75.88094133295287 disk.rbd0.in‚0 disk.md0.in‚ 9.26953125 disk.sdb2.in‚3.1875 disk.nvme0n1.in‚ 618.11328125 disk.sda1.in‚0 disk.dm-0.in‚ 1116.9140625! disk.dm-0.out‚101809.82421875 disk.sda2.in‚ 6.08203125 network.sent‚ 2.0480582e+07% disk./.usageGB‚2713.8700103759766 disk.sdb1.out‚0 disk.md0.out‚ 1024.44140625$ memory_percent‚92.10295677185059àÛnu”ñ"‚(:‰ ÜÉêÔ¼Ùñÿ gpu.0.gpu‚45.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚171.501' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.87525 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0s@iuò"‚(:ê ÜÉêÔ¾ÀŒ! disk.sdb2.out‚1064.3291015625 disk.dm-0.in‚ 1116.9140625 disk.sdb.in‚3.1875# proc.memory.rssMB‚ 3000.48046875 disk.sdb.out‚1064.3291015625 cpu‚25.201661383042016 disk.md0.out‚ 1025.83984375 disk.sdb2.in‚3.1875 disk.sda1.in‚0) disk./.usagePercent‚75.88093674654999 disk.sda1.out‚0! disk.dm-0.out‚102027.76171875# disk.nvme0n1.out‚52035.25390625 network.sent‚ 2.0486687e+07$ disk./.usageGB‚2713.869846343994 disk.rbd0.in‚0" disk.nvme1n1.out‚ 49992.5078125 disk.sdb1.in‚0) proc.memory.percent‚18.31347942352295 disk.sda2.in‚ 6.08203125 disk.rbd0.out‚ 2.466796875 disk.md0p1.in‚ 9.26953125 network.recv‚ 2.980932e+06! disk.sda2.out‚1064.3291015625 disk.nvme1n1.in‚ 498.80078125 disk.md0.in‚ 9.26953125 disk.rbd2.in‚0 disk.md0p1.out‚ 1025.83984375 disk.sda.in‚ 6.08203125$ memory_percent‚92.10143089294434 disk.sdb1.out‚0 disk.rbd2.out‚ 2.466796875 disk.nvme0n1.in‚ 618.11328125 disk.sda.out‚1064.3291015625( proc.memory.availableMB‚ 1294.1015625 proc.cpu.threads‚17”Ut¶’ó"‚(:‡ ëÉêÔåý‘þ gpu.0.gpu‚33.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚37.0 gpu.0.powerWatts‚169.61' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.4025 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ê¿…nô"‚(:ã ëÉêÔçϸÿ disk.sda1.out‚0 disk.sda2.in‚ 6.08203125 disk.rbd0.in‚0$ memory_percent‚92.10145473480225 disk.dm-0.in‚ 1116.9140625 disk.md0p1.out‚ 1027.78515625 disk.sdb.out‚ 1066.33203125 network.recv‚ 2.984973e+06) disk./.usagePercent‚75.88093845311852 disk.sda1.in‚0 disk.md0p1.in‚ 9.26953125" disk.nvme1n1.out‚ 50088.6640625# proc.memory.rssMB‚ 3000.48046875) proc.memory.percent‚18.31347942352295 proc.cpu.threads‚17 disk.nvme0n1.in‚ 618.11328125 disk.sdb1.in‚0 disk.md0.out‚ 1027.78515625 disk.sda2.out‚ 1066.33203125 disk.nvme1n1.in‚ 498.80078125 disk.rbd2.out‚ 2.466796875# disk.nvme0n1.out‚52297.31640625 disk.md0.in‚ 9.26953125 network.sent‚ 2.0492667e+07% disk./.usageGB‚2713.8699073791504 disk.sdb2.in‚3.1875! disk.dm-0.out‚102385.98046875 disk.sda.in‚ 6.08203125 disk.sdb1.out‚0) proc.memory.availableMB‚ 1294.09765625 disk.sdb2.out‚ 1066.33203125 disk.sdb.in‚3.1875 disk.sda.out‚ 1066.33203125 disk.rbd0.out‚ 2.466796875 disk.rbd2.in‚0 cpu‚25.20139948638914²]q©2õ"- _wandbruntime‚9294  _runtime‚9294)±gÆö"À & lr-AdamW/pg2‚0.0002659525987704483 trainer/global_step‚7999 $ _timestamp‚1.788519668630169e+09 _runtime‚9294.100633197 _step‚654 & lr-AdamW/pg1‚0.0002659525987704483k…GË÷"Å & lr-AdamW/pg2‚0.0002659525987704483 trainer/global_step‚7999 $ _timestamp‚1.788519668630169e+09 _runtime‚9294.100633197 _step‚654 & lr-AdamW/pg1‚0.0002659525987704483Ž0©=×2ø"- _wandbruntime‚9294  _runtime‚9294± !þ'ù"øO ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 203694080 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411420 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 164968982 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚657317728503296 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177047040 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1187 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 200608310 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 200582222 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9056165067965 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9089153295360 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9055267656545 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚616 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚745023792737792 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9489803227648 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 trainer/global_step‚7999 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚666406863795712 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 102582375 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 399863808 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 200608310 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚666406863795712 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1212 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410816 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 365577294 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 102582373 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 150529303 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚651731589170317 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007241 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚618 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955591168 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407872 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 227634176 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 164968982 _step‚655 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 164968984 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚657317710500352 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚47946312 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9489599533568 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚642675407062992 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450780 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917410816 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚666407781206528 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 365551204 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚666407781206528 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚745023565103616 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚642675424102352 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26088 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚735533989510144 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 200582222 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 150528685 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚735533965570048 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26090 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9090052703232 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚651730674719537 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955591168 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚47946928 % _timestamp‚1.7885196686314075e+09 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚657317710500352 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9089153295360 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26088 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955591168 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚47946928 % _timestamp‚1.7885196686314075e+09 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚657317710500352 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9089153295360 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26088 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9089153295360 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9055267656545 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚616 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚745023792737792 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9489803227648 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 trainer/global_step‚7999 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚666406863795712 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 102582375 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 399863808 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 200608310 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚666406863795712 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1212 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410816 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 365577294 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 102582373 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 150529303 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚651731589170317 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007241 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚618 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955591168 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407872 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 227634176 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 164968982 _step‚655 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 164968984 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚657317710500352 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚47946312 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9489599533568 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚642675407062992 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450780 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917410816 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚666407781206528 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 365551204 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚666407781206528 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚745023565103616 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚642675424102352 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26088 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚735533989510144 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 200582222 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 150528685 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚735533965570048 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26090”ÕÄ2û"- _wandbruntime‚9295  _runtime‚9295l]ß>ùü"‚N ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899421184 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚623 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 365621809 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 150547477 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚47952726 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9490981664256 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9490759109120 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177047040 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚651809763474983 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.a5îìllocated‚745114290395136 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 200606813 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9090266098688 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 200632927 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚735623308730880 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚666488524318720 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚657397340795904 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9090266098688 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 150546852 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚666488524318720 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917424128 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 102594751 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899421184 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31686 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411628 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 164988880 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚657397340795904 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 200632927 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607hž¸g 'ý"‡N B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899421184 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31686 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411628 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 164988880 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚657397340795904 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 200632927 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899421184 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚623 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 365621809 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 150547477 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚47952726 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9490981664256 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9490759109120 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177047040 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚651809763474983 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚745114290395136 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 200606813 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9090266098688 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 200632927 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚735623308730880 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚666488524318720 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚657397340795904 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9090266098688 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 150546852 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚666488524318720 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917424128 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 102594751 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2LoP2þ"- _wandbruntime‚9295  _runtime‚92959œx²Yÿ"Ó 0loss/FRt2/assignment_loss‚1.3980215231577555 1loss/SRqqt2/detection_loss‚1.0969209035195537 /loss/FBt4/detection_loss‚2.5812302112579344 1loss/FBt2/assignment_loss‚0.06653278552395207 1loss/FBt4/assignment_loss‚0.17760523382803511 .loss/FRt1/detection_loss‚2.113980843852016 2loss/SRqqt3/assignment_loss‚1.0208474041894078 /loss/FBt1/detection_loss‚2.1709075570106506 /loss/FBt2/detection_loss‚0.6600212802489599 _step‚657 1loss/SRqqt1/assignment_loss‚1.290456247714455 .loss/FRt3/detection_loss‚2.076449687051219 1loss/SRqqt1/detection_loss‚1.4829106560190681 %loss/total_loss‚37.34103694783023 /loss/FRt1/assignment_loss‚2.445954068348958 1loss/SRqqt2/assignment_loss‚1.007252595845168 1loss/FBt3/assignment_loss‚0.19092230701466584 /loss/FRt2/detection_loss‚1.8625273218585385 % _timestamp‚1.7885196695908225e+09 1loss/FBt1/assignment_loss‚0.17218053227648036 epoch‚6 /loss/FRt3/assignment_loss‚1.605585931345474 0loss/FRt4/assignment_loss‚2.8467468385185515 2loss/SRqqt4/assignment_loss‚1.9128438902122002 .loss/FRt4/detection_loss‚3.159867750746863 1loss/SRqqt3/detection_loss‚1.3581435726334652 trainer/global_step‚7999 0loss/SRqqt4/detection_loss‚2.584387997786204 .loss/FBt3/detection_loss‚2.058739807870653 _runtime‚ 9295.06100294Xš^€#Ø 1loss/FBt3/assignment_loss‚0.19092230701466584 /loss/FRt2/detection_loss‚1.8625273218585385 % _timestamp‚1.7885196695908225e+09 1loss/FBt1/assignment_loss‚0.17218053227648036 epoch‚6 /loss/FRt3/assignment_loss‚1.605585931345474 0loss/FRt4/assignment_loss‚2.8467468385185515 2loss/SRqqt4/assignment_loss‚1.9128438902122002 .loss/FRt4/detection_loss‚3.159867750746863 1loss/SRqqt3/detection_loss‚1.3581435726334652 trainer/global_step‚7999 0loss/SRqqt4/detection_loss‚2.584387997786204 .loss/FBt3/detection_loss‚2.058739807870653 _runtime‚ 9295.06100294 0loss/FRt2/assignment_loss‚1.3980215231577555 1loss/SRqqt2/detection_loss‚1.0969209035195537 /loss/FBt4/detection_loss‚2.5812302112579344 1loss/FBt2/assignment_loss‚0.06653278552395207 1loss/FBt4/assignment_loss‚0.17760523382803511 .loss/FRt1/detection_loss‚2.113980843852016 2loss/SRqqt3/assignment_loss‚1.0208474041894078 /loss/FBt1/detection_loss‚2.1709075570106506 /loss/FBt2/detection_loss‚0.6600212802489599 _step‚657 1loss/SRqqt1/assignment_loss‚1.290456247714455 .loss/FRt3/detection_loss‚2.076449687051219 1loss/SRqqt1/detection_loss‚1.4829106560190681 %loss/total_loss‚37.34103694783023 /loss/FRt1/assignment_loss‚2.445954068348958 1loss/SRqqt2/assignment_loss‚1.007252595845168‘D£_‘”#‚(:‰ úÉêÔÐæ¢þ gpu.0.gpu‚45.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚169.333' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.33325 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0mPÖw‚#‚(:ì úÉêÔËšØþ disk.sdb2.in‚3.1875# disk.nvme0n1.out‚52385.66796875 disk.md0.out‚ 1029.29296875$ memory_percent‚92.10314750671387( proc.memory.availableMB‚ 1293.8203125 network.recv‚ 2.989797e+06 disk.md0p1.in‚ 9.26953125 disk.sdb.in‚3.1875 disk.sda.in‚ 6.08203125 disk.rbd0.in‚0$ disk./.usageGB‚2712.815849304199# disk.nvme1n1.out‚50208.57421875 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.md0.in‚ 9.26953125# proc.memory.rssMB‚ 3000.48046875 proc.cpu.threads‚17( disk./.usagePercent‚75.8514665481853 disk.sda1.in‚0 disk.dm-0.in‚ 1116.9140625" disk.sda2.out‚1067.88623046875 disk.dm-0.out‚102594.2421875 disk.md0p1.out‚ 1029.29296875 disk.nvme1n1.in‚ 498.80078125 network.sent‚ 2.0499096e+07 disk.nvme0n1.in‚ 618.11328125! disk.sdb.out‚1067.88623046875 disk.rbd2.in‚0" disk.sdb2.out‚1067.88623046875 disk.sda1.out‚0) proc.memory.percent‚18.31347942352295 disk.rbd2.out‚ 2.466796875! disk.sda.out‚1067.88623046875 disk.sda2.in‚ 6.08203125 disk.rbd0.out‚ 2.466796875 cpu‚25.20110660494076@ÔfZ”ƒ#‚(:‰ ‰ÊêÔžÄþ gpu.0.gpu‚55.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚173.501' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.37525 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¼|ù{„#‚(:ð ‰ÊêÔøЕÿ disk.dm-0.in‚ 1116.9140625 disk.md0p1.in‚ 9.26953125 disk.md0p1.out‚ 1030.99609375! disk.sda.out‚1069.65478515625 disk.nvme1n1.in‚ 498.80078125 disk.rbd2.out‚ 2.466796875% disk./.usageGB‚2712.8157920837402 disk.sdb1.out‚0 disk.md0.in‚ 9.26953125 disk.rbd2.in‚0" disk.sdb2.out‚1069.65478515625 network.recv‚ 2.994611e+06! disk.dm-0.out‚102630.40234375# disk.nvme0n1.out‚52385.72265625 disk.sdb.in‚3.1875 disk.rbd0.out‚ 2.466796875 disk.sdb2.in‚3.1875 disk.sda1.in‚0 disk.sda1.out‚0 network.sent‚ 2.0606097e+07) disk./.usagePercent‚75.85146494827733 disk.nvme0n1.in‚ 618.11328125" disk.sda2.out‚1069.65478515625 proc.cpu.threads‚17! disk.sdb.out‚1069.65478515625 disk.sda.in‚ 6.08203125 disk.sdb1.in‚0$ memory_percent‚92.10317134857178) proc.memory.availableMB‚ 1293.81640625# proc.memory.rssMB‚ 3000.48046875) proc.memory.percent‚18.31347942352295" disk.nvme1n1.out‚ 50244.6796875 disk.md0.out‚ 1030.99609375 cpu‚25.200780984565114 disk.sda2.in‚ 6.08203125 disk.rbd0.in‚0ñ2Ïò“…#‚(:ˆ ˜ÊêÔ•ØÈý gpu.0.gpu‚56.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚37.0 gpu.0.powerWatts‚152.282' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚38.0705 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÿÍ Jv†#‚(:ë ˜ÊêÔ‘ââþ% disk./.usageGB‚2712.2860069274902 disk.sdb2.in‚3.1875 disk.dm-0.in‚ 1116.9140625 disk.sda.in‚ 6.08203125# disk.nvme1n1.out‚50352.80078125 network.recv‚ 2.999468e+06) disk./.usagePercent‚75.83665193357626# disk.nvme0n1.out‚52385.84765625! disk.sda.out‚1071.59033203125 disk.rbd2.in‚0 disk.sda1.out‚0 disk.md0.in‚ 9.26953125$ memory_percent‚92.10622310638428 cpu‚25.200514701226922 network.sent‚ 2.061215e+07 disk.sda1.in‚0 disk.dm-0.out‚102738.6484375 disk.md0p1.out‚ 1032.859375# proc.memory.rssMB‚ 3000.48046875) proc.memory.percent‚18.31347942352295 proc.cpu.threads‚17 disk.nvme0n1.in‚ 618.11328125 disk.sdb.in‚3.1875 disk.sdb1.in‚0 disk.md0p1.in‚ 9.26953125! disk.sdb.out‚1071.59033203125" disk.sda2.out‚1071.59033203125 disk.rbd0.in‚0 disk.sdb1.out‚0 disk.rbd2.out‚ 2.466796875" disk.sdb2.out‚1071.59033203125 disk.sda2.in‚ 6.08203125 disk.rbd0.out‚ 2.466796875 disk.nvme1n1.in‚ 498.80078125) proc.memory.availableMB‚ 1293.31640625 disk.md0.out‚ 1032.859375C!£4“‡#‚(:ˆ §ÊêÔÉÌúþ gpu.0.gpu‚40.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚165.266' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚41.3165 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ᆩgˆ#‚(:Ü §ÊêÔ·•çÿ proc.cpu.threads‚17 disk.rbd0.out‚ 2.466796875# disk.nvme1n1.out‚50413.50390625 disk.sdb1.out‚0 disk.rbd2.out‚ 2.466796875 disk.dm-0.in‚ 1116.9140625 disk.sdb.in‚3.1875) proc.memory.percent‚18.31347942352295 network.recv‚ 3.003642e+06) disk./.usagePercent‚75.83410200023482$ disk./.usageGB‚2712.194808959961 disk.sda2.in‚ 6.08203125 disk.sdb1.in‚0 disk.rbd2.in‚0 disk.sdb2.out‚ 1073.20703125$ memory_percent‚92.10178852081299 disk.sda.in‚ 6.08203125 disk.nvme1n1.in‚ 498.80078125) proc.memory.availableMB‚ 1294.04296875# proc.memory.rssMB‚ 3000.48046875 disk.rbd0.in‚0 disk.sdb2.in‚3.1875" disk.nvme0n1.out‚ 52386.8203125 cpu‚25.20026791245961 disk.md0p1.in‚ 9.26953125 disk.nvme0n1.in‚ 618.11328125 disk.sda1.in‚0 disk.sda1.out‚0 disk.sdb.out‚ 1073.20703125 disk.sda.out‚ 1073.20703125 disk.sda2.out‚ 1073.20703125 disk.md0.in‚ 9.26953125 disk.md0.out‚ 1034.40625! disk.dm-0.out‚102800.32421875 network.sent‚ 2.0620184e+07 disk.md0p1.out‚ 1034.40625Gš\72‰#- _wandbruntime‚9346  _runtime‚9346¼x¿"ÉŠ#à ' lr-AdamW/pg1‚0.00026024284531324306 ' lr-AdamW/pg2‚0.00026024284531324306 trainer/global_step‚8049 % _timestamp‚1.7885197208280604e+09 _runtime‚9346.298456371 _step‚658ˆ²Õè΋#È ' lr-AdamW/pg1‚0.00026024284531324306 ' lr-AdamW/pg2‚0.00026024284531324306 trainer/global_step‚8049 % _timestamp‚1.7885197208280604e+09 _runtime‚9346.298456371 _step‚658’Û÷è2Œ#- _wandbruntime‚9346  _runtime‚9346?.e#÷O XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚661299225277952 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1212 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9111611725073 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9110714313653 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 165963884 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 165963884 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450780 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 $ _timestamp‚1.788519720829137e+09 _step‚659 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 201815124 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 399863808 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚739999926606848 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚655686040533825 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚661299243280896 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 165963882 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26088 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 220471296 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214665 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411420 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9145707283456 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚670444033153536 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1187 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9547928571904 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚640 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 244411392 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚749548099590144 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955591168 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 151436493 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407872 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚642 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9144807875584 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 201841212 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 103201175 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 201841212 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 trainer/global_step‚8049 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 367779006 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26088 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 103201173 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955591168 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚739999950546944 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26090 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9145707283456 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 151437135 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚670444033153536 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚661299243280896 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177047040 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 165963884 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450780 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 $ _timestamp‚1.788519720829137e+09 _step‚659 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 201815124 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 399863808 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚739999926606848 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚655686040533825 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚661299243280896 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 165963882 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26088 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 220471296 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214665 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411420 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9145707283456 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚670444033153536 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1187 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9547928571904 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚640 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 244411392 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚749548099590144 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955591168 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 151436493 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407872 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚642 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9144807875584 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 201841212 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 103201175 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 201841212 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 trainer/global_step‚8049 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 367779006 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26088 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 103201173 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955591168 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚739999950546944 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26090 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9145707283456 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 151437135 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚670444033153536 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚661299243280896 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177047040 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1212 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9111611725073 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9110714313653 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171“°2#- _wandbruntime‚9347  _runtime‚9347ò£~°'#‚N UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 _step‚660 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9145920678912 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 367849611 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 201865829 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚617 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 201865829 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚615 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚740089245827584 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899421184 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177047040 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411628 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 367823495 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955591168 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 % _timestamp‚1.7885197218031335e+09 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26114 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚670525693676544 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 165983780 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 103213551 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177047040 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9111822974971 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450988 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 201839715 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9549320226816 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚670525693676544 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9145920678912 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 367823495 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31686 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9146820100096 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚740089269767680 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 151454530 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚749638589994496 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 244398080 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9549320226816 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚670525693676544 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9145920678912 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 367823495 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31686 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9146820100096 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚740089269767680 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 151454530 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚749638589994496 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 244398080 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9145920678912 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 367849611 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 201865829 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚617 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 201865829 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚615 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚740089245827584 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899421184 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177047040 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411628 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 367823495 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955591168 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 % _timestamp‚1.7885197218031335e+09 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26114 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚670525693676544 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 165983780 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 103213551 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177047040 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9111822974971 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_poo˜& ]l.peak‚ 12835443712 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450988 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0”€WýV2’#- _wandbruntime‚9347  _runtime‚9347Ìžƒ»[“#Õ trainer/global_step‚8049 0loss/FRt2/assignment_loss‚1.3457571230036147 0loss/FBt2/assignment_loss‚0.0972742652606712 1loss/FBt3/assignment_loss‚0.07089968935805599 /loss/FRt3/detection_loss‚2.4413859346373514 1loss/SRqqt3/detection_loss‚1.4723207789162795 .loss/FBt4/detection_loss‚3.614336109161377 1loss/SRqqt1/assignment_loss‚1.278610608625141 1loss/FBt1/assignment_loss‚0.09848178829940935 .loss/FRt4/detection_loss‚3.089694184285623 /loss/FBt2/detection_loss‚0.5450991803198122 %loss/total_loss‚41.07622389559484 3loss/FBt4/assignment_loss‚0.0014989759187301389 /loss/FRt1/detection_loss‚1.9878518274816452 0loss/SRqqt4/detection_loss‚2.813776385325652 /loss/FBt1/detection_loss‚2.2446247339248657 epoch‚6 2loss/SRqqt2/assignment_loss‚0.9304474276579815 2loss/SRqqt4/assignment_loss‚1.7932793658513289 _runtime‚9347.276356979 /loss/FRt1/assignment_loss‚3.134331669807434 2loss/SRqqt3/assignment_loss‚1.4706139610735354 0loss/SRqqt1/detection_loss‚1.645141159776937 /loss/FRt4/assignment_loss‚3.244305866735953 1loss/SRqqt2/detection_loss‚0.8985834044901884 .loss/FBt3/detection_loss‚2.365475747320387 _step‚661 0loss/FRt3/assignment_loss‚2.7595879665829917 /loss/FRt2/detection_loss‚1.7328457417798804 $ _timestamp‚1.788519721806172e+09Š°—î`”#Ú trainer/global_step‚8049 0loss/FRt2/assignment_loss‚1.3457571230036147 0loss/FBt2/assignment_loss‚0.0972742652606712 1loss/FBt3/assignment_loss‚0.07089968935805599 /loss/FRt3/detection_loss‚2.4413859346373514 1loss/SRqqt3/detection_loss‚1.4723207789162795 .loss/FBt4/detection_loss‚3.614336109161377 1loss/SRqqt1/assignment_loss‚1.278610608625141 1loss/FBt1/assignment_loss‚0.09848178829940935 .loss/FRt4/detection_loss‚3.089694184285623 /loss/FBt2/detection_loss‚0.5450991803198122 %loss/total_loss‚41.07622389559484 3loss/FBt4/assignment_loss‚0.0014989759187301389 /loss/FRt1/detection_loss‚1.9878518274816452 0loss/SRqqt4/detection_loss‚2.813776385325652 /loss/FBt1/detection_loss‚2.2446247339248657 epoch‚6 2loss/SRqqt2/assignment_loss‚0.9304474276579815 2loss/SRqqt4/assignment_loss‚1.7932793658513289 _runtime‚9347.276356979 /loss/FRt1/assignment_loss‚3.134331669807434 2loss/SRqqt3/assignment_loss‚1.4706139610735354 0loss/SRqqt1/detection_loss‚1.645141159776937 /loss/FRt4/assignment_loss‚3.244305866735953 1loss/SRqqt2/detection_loss‚0.8985834044901884 .loss/FBt3/detection_loss‚2.365475747320387 _step‚661 0loss/FRt3/assignment_loss‚2.7595879665829917 /loss/FRt2/detection_loss‚1.7328457417798804 $ _timestamp‚1.788519721806172e+09•|Ó”•#‚(:‰ ¶ÊêÔÏà´þ gpu.0.gpu‚55.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚164.039' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚41.00975 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‰){–#‚(:ð ¶ÊêÔüÅÿ disk.sda2.in‚ 6.08203125 network.sent‚ 2.0637246e+07" disk.sda2.out‚1074.95849609375 disk.rbd0.out‚ 2.466796875" disk.nvme1n1.out‚ 50414.4921875 disk.nvme0n1.in‚ 618.11328125) proc.memory.percent‚18.31347942352295) disk./.usagePercent‚75.83434753277977$ memory_percent‚92.10789203643799 disk.md0p1.out‚ 1036.09765625# disk.nvme0n1.out‚52459.03515625 network.recv‚ 3.008675e+06 disk.rbd0.in‚0 disk.md0p1.in‚ 9.26953125% disk./.usageGB‚2712.2035903930664! disk.sda.out‚1074.95849609375 disk.nvme1n1.in‚ 498.80078125 disk.sdb1.in‚0 disk.md0.in‚ 9.26953125 disk.md0.out‚ 1036.09765625 disk.sdb.in‚3.1875 proc.cpu.threads‚17 disk.rbd2.in‚0! disk.dm-0.out‚102873.52734375) proc.memory.availableMB‚ 1293.04296875 disk.rbd2.out‚ 2.466796875 cpu‚25.199949407257364 disk.sdb1.out‚0 disk.sdb2.in‚3.1875" disk.sdb2.out‚1074.95849609375 disk.sda1.in‚0 disk.sda1.out‚0! disk.sdb.out‚1074.95849609375# proc.memory.rssMB‚ 3000.48046875 disk.dm-0.in‚ 1116.9140625 disk.sda.in‚ 6.08203125a2-÷ž—#‚(:“ ÅÊêÔœ¢’ÿ gpu.0.gpu‚50.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚37.0 gpu.0.powerWatts‚127.106' gpu.0.enforcedPowerLimitWatts‚400.0) gpu.0.powerPercent‚31.776499999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÞôÖ©u˜#‚(:ê ÅÊêÔµ»Óÿ disk.md0p1.out‚ 1037.49609375 network.sent‚ 2.0731167e+07 disk.nvme0n1.in‚ 618.11328125$ memory_percent‚92.10488796234131) proc.memory.percent‚18.31347942352295! disk.dm-0.out‚102965.52734375$ disk./.usageGB‚2712.364082336426 disk.sdb1.out‚0 disk.md0.in‚ 9.26953125 disk.md0.out‚ 1037.49609375 disk.rbd2.in‚0) proc.memory.availableMB‚ 1293.53515625 proc.cpu.threads‚17 disk.sda2.in‚ 6.08203125! disk.sda2.out‚1076.4267578125 disk.rbd0.in‚0" disk.nvme1n1.out‚ 50414.6328125 disk.sda1.in‚0 disk.sda1.out‚0 disk.sdb.in‚3.1875 disk.rbd2.out‚ 2.466796875 disk.sdb.out‚1076.4267578125 disk.md0p1.in‚ 9.26953125 network.recv‚ 3.012624e+06 disk.sda.out‚1076.4267578125 disk.nvme1n1.in‚ 498.80078125 cpu‚25.199682844265322( disk./.usagePercent‚75.8388349546872 disk.sda.in‚ 6.08203125 disk.rbd0.out‚ 2.466796875 disk.sdb2.in‚3.1875! disk.sdb2.out‚1076.4267578125 disk.dm-0.in‚ 1116.9140625# disk.nvme0n1.out‚52550.89453125# proc.memory.rssMB‚ 3000.48046875 disk.sdb1.in‚0ˆª[®’™#‚(:‡ ÔÊêÔ—º½ÿ gpu.0.gpu‚53.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚162.204' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚40.551 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ᙕQlš#‚(:á ÔÊêÔ±œñÿ disk.sda1.in‚0 disk.md0.in‚ 9.26953125 disk.rbd2.in‚0 disk.nvme0n1.in‚ 618.11328125 disk.nvme0n1.out‚ 52626.1875 disk.sdb.in‚3.1875# proc.memory.rssMB‚ 3000.48046875 disk.sdb1.out‚0 disk.md0.out‚ 1039.359375! disk.sdb.out‚1078.33740234375) proc.memory.percent‚18.31347942352295 disk.md0p1.out‚ 1039.359375 disk.nvme1n1.out‚ 50415.09375! disk.sda.out‚1078.33740234375 network.sent‚ 2.0736374e+07 disk.rbd0.in‚0 disk.sdb1.in‚0" disk.sdb2.out‚1078.33740234375$ memory_percent‚92.10188388824463) proc.memory.availableMB‚ 1294.02734375 disk.rbd2.out‚ 2.466796875 disk.md0p1.in‚ 9.26953125 proc.cpu.threads‚17 disk.sda1.out‚0 disk.dm-0.in‚ 1116.9140625 cpu‚25.19941652756612 disk.dm-0.out‚ 103041.28125) disk./.usagePercent‚75.83902705030555$ disk./.usageGB‚2712.370952606201 disk.sda.in‚ 6.08203125 disk.sda2.in‚ 6.08203125" disk.sda2.out‚1078.33740234375 disk.rbd0.out‚ 2.466796875 disk.nvme1n1.in‚ 498.80078125 network.recv‚ 3.016712e+06 disk.sdb2.in‚3.1875#ˆ‰2›#- _wandbruntime‚9398  _runtime‚9398ü»÷¤Éœ#à ' lr-AdamW/pg1‚0.00025456822939692197 ' lr-AdamW/pg2‚0.00025456822939692197 trainer/global_step‚8099 % _timestamp‚1.7885197734791179e+09 _runtime‚9398.949550549 _step‚662Á&yÎ#È % _timestamp‚1.7885197734791179e+09 _runtime‚9398.949550549 _step‚662 ' lr-AdamW/pg1‚0.00025456822939692197 ' lr-AdamW/pg2‚0.00025456822939692197 trainer/global_step‚8099–¿¶V2ž#- _wandbruntime‚9398  _runtime‚9398ö1É‘ý'Ÿ#÷O QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410816 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚659640491897333 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 103819975 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 _step‚663 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 370032898 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 399863808 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 166958784 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26088 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 370006808 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9606466647040 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 166958782 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 203048026 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 203074114 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214665 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917410816 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 152344328 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9606235689984 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 203048026 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1212 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 $ _timestamp‚1.788519773480254e+09 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 166958784 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚48524355 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9167058382181 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _runtime‚9398.950708704 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚48524999 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1187 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚665280758058496 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 230957056 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955591168 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411420 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407872 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚744465887643648 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚674481202511360 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚665280740055552 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177047040 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9201361863680 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9200462455808 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 254897152 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚650473416475792 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 103819973 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚754072123333632 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450780 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚650473433515152 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚665280758058496 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚646 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955591168 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26090 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚674482119922176 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚744465911583744 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚674481202511360 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚644 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 trainer/global_step‚8099 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9201361863680 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 103819975 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 _step‚663 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 370032898 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 399863808 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 166958784 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26088 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 370006808 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9606466647040 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 166958782 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 203048026 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 203074114 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214665 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917410816 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 152344328 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9606235689984 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 203048026 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1212 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 $ _timestamp‚1.788519773480254e+09 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 166958784 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚48524355 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9167058382181 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _runtime‚9398.950708704 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚48524999 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1187 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚665280758058496 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 230957056 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955591168 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411420 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407872 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚744465887643648 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚674481202511360 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚665280740055552 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177047040 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9201361863680 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9200462455808 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 254897152 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚650473416475792 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 103819973 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚754072123333632 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450780 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚650473433515152 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚665280758058496 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚646 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955591168 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26090 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚674482119922176 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚744465911583744 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚674481202511360 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚644 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 trainer/global_step‚8099 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9201361863680 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31686 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 166978680 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899421184 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899421184 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚754162851191296 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450988 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9202474680320 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 152363096 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9607620386816 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 203072617 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 203072617 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚659719580652987 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26114 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _runtime‚9400.078153327 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 370077413 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9201575259136 trainer/global_step‚8099 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177047040 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚674562863034368 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917424128 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177047040 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 203098731 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚659718666201999 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚674561945610240 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1187 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 166978682 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 370051297 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚619 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚48530128 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 152362477 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚665360370351104 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 203098731 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9202474680320 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 103832351 $ _timestamp‚1.788519774606904e+09 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9607399928832 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚674562863034368 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚665360370351104 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9201575259136 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 370077413 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26114 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 166978682ŸAt '£#†N E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _runtime‚9400.078153327 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 370077413 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9201575259136 trainer/global_step‚8099 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177047040 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚674562863034368 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917424128 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177047040 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 203098731 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚659718666201999 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚674561945610240 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1187 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 166978682 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 370051297 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚619 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚48530128 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 152362477 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚665360370351104 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 203098731 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9202474680320 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 103832351 $ _timestamp‚1.788519774606904e+09 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9607399928832 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚674562863034368 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚665360370351104 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9201575259136 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 370077413 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26114 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 166978682 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚650551413609280 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 166978680 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31686 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 166978680 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899421184 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899421184 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚754162851191296 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450988 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9202474680320 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 152363096 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9607620386816 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 203072617 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 203072617 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚659719580652987 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26114 PDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚678518371869184 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚48812638 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚669262254833152 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9222505039289 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917410816 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177047040 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚669262272836096 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚669262272836096 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410816 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚758596520762880 _step‚667 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚678518371869184 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 204307016 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9256117036032 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 167953684 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚748931872620544 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9664412990976 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚654372438221552 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 104438773 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26088 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 104438775 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1187 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9256117036032 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 399863808 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9664648142336 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26090 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9257016443904 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9221607627869 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955591168 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 153250765 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚648 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 259091456 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450780 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚678519289280000 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 204307016 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 167953684 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚758596261671424 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214665B_e(´#ûO QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 204280928 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 167953682 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 399863808 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9664648142336 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26090 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9257016443904 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9221607627869 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955591168 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 153250765 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚648 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 259091456 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450780 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚678519289280000 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 204307016 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 167953684 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚758596261671424 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214665 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 204280928 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407872 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚748931848680448 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚678518371869184 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚48812638 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚669262254833152 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9222505039289 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917410816 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177047040 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚669262272836096 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚669262272836096 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410816 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚758596520762880 _step‚667 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚678518371869184 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 204307016 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9256117036032 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 167953684 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚748931872620544 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9664412990976 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚654372438221552 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 104438773 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26088 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 104438775 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1187 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9256117036032 JDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9257229839360 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 204331633 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 204305519 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26114 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 153268842 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 259078144 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 _step‚668 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚653 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9258129260544 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 153269497 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚663673117565507 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 167973580 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31686 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚48818346 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9665560403456 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚654450401276320 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 167973582 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚669341903131648 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 372279099 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9258129260544 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 167973582 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 104451151 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 104451149 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9222716289187 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 372305215 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚749021167901184 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007241 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 372279099 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177047040 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚669341885128704 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 235138048 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 $ _timestamp‚1.788519830827373e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚758686987382784 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚678599114968064 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 trainer/global_step‚8149 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955591168 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚669341885128704 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899421184 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚678600032392192 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9223613700815 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9257229839360 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 204331633 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚669341903131648 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1187 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚663674032016495 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 167973580 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚749021191841280 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917424128 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411628 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0ÂÁ9 b$·#†N I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26114 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 153268842 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 259078144 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 _step‚668 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚653 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9258129260544 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 153269497 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚663673117565507 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 167973580 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31686 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚48818346 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9665560403456 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚654450401276320 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 167973582 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚669341903131648 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 372279099 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9258129260544 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 167973582 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 104451151 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 104451149 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9222716289187 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 372305215 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚749021167901184 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007241 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 372279099 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177047040 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚669341885128704 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 235138048 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 $ _timestamp‚1.788519830827373e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚758686987382784 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚678599114968064 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 trainer/global_step‚8149 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955591168 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚669341885128704 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899421184 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚678600032392192 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9223613700815 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9257229839360 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 204331633 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚669341903131648 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1187 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚663674032016495 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 167973580 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚749021191841280 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917424128 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411628 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UEDeviceStatsMonitor.on_tŠÕ"ãªrain_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9257229839360 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 204331633 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 204305519 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2œˆMâ,2¸#- _wandbruntime‚9456  _runtime‚9456c§Y¹#Ó 2loss/SRqqt3/assignment_loss‚1.1731730874019617 1loss/SRqqt4/detection_loss‚2.4108867336202553 /loss/FBt2/detection_loss‚0.5524787776698561 % _timestamp‚1.7885198308292968e+09 _runtime‚9456.299450041 0loss/FRt3/assignment_loss‚1.8804602197238378 0loss/FBt4/assignment_loss‚0.7230054415267659 trainer/global_step‚8149 .loss/FRt4/assignment_loss‚2.44758078828454 2loss/SRqqt4/assignment_loss‚1.8917702223967623 1loss/FBt2/assignment_loss‚0.12448177448281746 /loss/FRt2/detection_loss‚1.5224649474809044 .loss/FRt3/detection_loss‚2.501669051817485 .loss/FRt4/detection_loss‚2.948561990633607 /loss/FBt3/detection_loss‚2.9935403615236282 2loss/SRqqt1/assignment_loss‚1.3117430783869302 _step‚669 1loss/SRqqt2/detection_loss‚1.1000745227200943 /loss/FBt4/detection_loss‚2.5114844540754953 0loss/FRt2/assignment_loss‚1.3739691094348305 1loss/FBt3/assignment_loss‚0.10379001387627795 %loss/total_loss‚39.21986073913331 /loss/FRt1/assignment_loss‚2.248226367102729 2loss/SRqqt2/assignment_loss‚0.9877205639015556 .loss/FRt1/detection_loss‚2.379507400300988 0loss/SRqqt1/detection_loss‚1.579572297388413 1loss/SRqqt3/detection_loss‚1.6824601678823938 0loss/FBt1/assignment_loss‚0.2521331169583181 .loss/FBt1/detection_loss‚2.519106250542861 epoch‚6k¦€i^º#Ø /loss/FRt1/assignment_loss‚2.248226367102729 2loss/SRqqt2/assignment_loss‚0.9877205639015556 .loss/FRt1/detection_loss‚2.379507400300988 0loss/SRqqt1/detection_loss‚1.579572297388413 1loss/SRqqt3/detection_loss‚1.6824601678823938 0loss/FBt1/assignment_loss‚0.2521331169583181 .loss/FBt1/detection_loss‚2.519106250542861 epoch‚6 2loss/SRqqt3/assignment_loss‚1.1731730874019617 1loss/SRqqt4/detection_loss‚2.4108867336202553 /loss/FBt2/detection_loss‚0.5524787776698561 % _timestamp‚1.7885198308292968e+09 _runtime‚9456.299450041 0loss/FRt3/assignment_loss‚1.8804602197238378 0loss/FBt4/assignment_loss‚0.7230054415267659 trainer/global_step‚8149 .loss/FRt4/assignment_loss‚2.44758078828454 2loss/SRqqt4/assignment_loss‚1.8917702223967623 1loss/FBt2/assignment_loss‚0.12448177448281746 /loss/FRt2/detection_loss‚1.5224649474809044 .loss/FRt3/detection_loss‚2.501669051817485 .loss/FRt4/detection_loss‚2.948561990633607 /loss/FBt3/detection_loss‚2.9935403615236282 2loss/SRqqt1/assignment_loss‚1.3117430783869302 _step‚669 1loss/SRqqt2/detection_loss‚1.1000745227200943 /loss/FBt4/detection_loss‚2.5114844540754953 0loss/FRt2/assignment_loss‚1.3739691094348305 1loss/FBt3/assignment_loss‚0.10379001387627795 %loss/total_loss‚39.21986073913331|L”»#‚(:‰ ŸËêÔî¥Ýý gpu.0.gpu‚45.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚37.0 gpu.0.powerWatts‚166.825' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚41.70625 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Üý¦'{¼#‚(:ð ŸËêÔ„šÚÿ disk.md0.out‚ 1047.39453125 disk.rbd2.in‚0 network.sent‚ 2.0867374e+07 network.recv‚ 3.038204e+06$ memory_percent‚92.10479259490967) proc.memory.availableMB‚ 1293.55078125# proc.memory.rssMB‚ 3000.48046875) proc.memory.percent‚18.31347942352295) disk./.usagePercent‚75.86622185953848! disk.dm-0.out‚103826.54296875 disk.rbd0.out‚ 2.478515625" disk.nvme1n1.out‚ 50416.6328125 disk.md0.in‚ 9.26953125 disk.sda1.out‚0 disk.nvme1n1.in‚ 498.80078125 disk.sda.in‚ 6.08203125 cpu‚25.198105010569595 disk.rbd2.out‚ 2.541015625 disk.md0p1.in‚ 9.26953125# disk.nvme0n1.out‚53409.91015625 disk.sdb.in‚3.1875 disk.rbd0.in‚0 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.sda1.in‚0 disk.sdb2.in‚3.1875" disk.sdb2.out‚1086.67724609375 proc.cpu.threads‚17 disk.dm-0.in‚ 1116.9140625 disk.nvme0n1.in‚ 618.11328125! disk.sdb.out‚1086.67724609375 disk.sda2.in‚ 6.08203125% disk./.usageGB‚2713.3435707092285 disk.md0p1.out‚ 1047.39453125! disk.sda.out‚1086.67724609375" disk.sda2.out‚1086.67724609375ë¡Ô´’½#‚(:‡ ®ËêÔšËÖÿ gpu.0.gpu‚54.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚162.204' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚40.551 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0šªWqo¾#‚(:ä ®ËêÔçÎ disk.md0.in‚ 9.26953125 disk.md0p1.out‚ 1048.63671875# proc.memory.rssMB‚ 3000.48046875 network.sent‚ 2.0975593e+07 disk.sdb2.in‚3.1875 disk.sda2.in‚ 6.08203125 disk.sda1.in‚0 disk.dm-0.in‚ 1116.9140625 disk.md0p1.in‚ 9.26953125$ memory_percent‚92.10319519042969) disk./.usagePercent‚75.88020206880475 disk.rbd0.out‚ 2.541015625 disk.nvme1n1.in‚ 498.80078125 disk.sdb1.in‚0 disk.md0.out‚ 1048.63671875 disk.rbd2.in‚0 disk.dm-0.out‚ 104075.703125! disk.sdb.out‚1087.97314453125 network.recv‚ 3.044269e+06% disk./.usageGB‚2713.8435707092285 disk.rbd2.out‚ 2.6201171875 disk.sda1.out‚0" disk.sda2.out‚1087.97314453125% proc.memory.availableMB‚ 1293.8125 cpu‚25.19786359042731 proc.cpu.threads‚17" disk.sdb2.out‚1087.97314453125 disk.sda.in‚ 6.08203125! disk.sda.out‚1087.97314453125) proc.memory.percent‚18.31347942352295 disk.rbd0.in‚0! disk.nvme1n1.out‚ 50416.828125 disk.sdb1.out‚0 disk.nvme0n1.out‚ 53658.875 disk.sdb.in‚3.1875 disk.nvme0n1.in‚ 618.11328125Sî“¿#‚(:ˆ ½ËêÔñ£üÿ gpu.0.gpu‚31.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚167.774' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚41.9435 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0wŒhÀ#‚(:Ý ½ËêÔ¾öî€ disk.rbd2.in‚0$ memory_percent‚92.10476875305176 disk.rbd0.in‚0! disk.sda.out‚1089.91064453125 disk.nvme1n1.out‚ 50417.03125" disk.sdb2.out‚1089.91064453125 disk.md0p1.out‚1050.5 disk.sdb2.in‚3.1875 disk.dm-0.out‚104255.9921875( proc.memory.availableMB‚ 1293.5546875 disk.sdb.in‚3.1875 disk.nvme1n1.in‚ 498.80078125 disk.md0.out‚1050.5 disk.sda1.out‚0" disk.sda2.out‚1089.91064453125" disk.nvme0n1.out‚ 53859.7109375# proc.memory.rssMB‚ 3000.48046875 proc.cpu.threads‚17 disk.rbd0.out‚ 2.541015625 disk.rbd2.out‚ 2.6201171875 disk.sda1.in‚0 disk.sdb1.in‚0 disk.md0.in‚ 9.26953125 disk.dm-0.in‚ 1116.9140625) proc.memory.percent‚18.31347942352295 disk.md0p1.in‚ 9.26953125 disk.sdb1.out‚0 disk.sda2.in‚ 6.08203125 network.sent‚ 2.0980906e+07 network.recv‚ 3.047653e+06) disk./.usagePercent‚75.88020217546529$ disk./.usageGB‚2713.843574523926 disk.nvme0n1.in‚ 618.11328125 cpu‚25.197631939467186! disk.sdb.out‚1089.91064453125 disk.sda.in‚ 6.08203125p*Z$”Á#‚(:‰ ÌËêÔ‚áùý gpu.0.gpu‚52.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚164.989' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚41.24725 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚04ù]_Â#‚(:Ô ÌËêÔÇ¡‹ÿ disk.nvme1n1.out‚ 50417.625 disk.sdb2.out‚ 1091.7890625 disk.sdb.out‚ 1091.7890625 network.recv‚ 3.05206e+06! disk.dm-0.out‚104526.21484375 disk.nvme0n1.in‚ 618.11328125) proc.memory.percent‚18.31347942352295 disk.md0.in‚ 9.26953125 disk.md0.out‚ 1052.3125 disk.rbd2.out‚ 2.6826171875 disk.dm-0.in‚ 1116.9140625 disk.sda.in‚ 6.08203125$ memory_percent‚92.10631847381592% disk./.usageGB‚2713.8436317443848 disk.sdb2.in‚3.1875# disk.nvme0n1.out‚54108.58984375 disk.rbd0.out‚ 2.6201171875# proc.memory.rssMB‚ 3000.48046875 proc.cpu.threads‚17) disk./.usagePercent‚75.88020377537326 disk.nvme1n1.in‚ 498.80078125 network.sent‚ 2.0986908e+07 disk.sdb1.in‚0 disk.sda1.out‚0 disk.sda.out‚ 1091.7890625 disk.sda2.in‚ 6.08203125 disk.sdb1.out‚0 disk.sda1.in‚0 disk.sdb.in‚3.1875 disk.rbd0.in‚0 cpu‚25.19735704302703 disk.rbd2.in‚0 disk.md0p1.in‚ 9.26953125 disk.md0p1.out‚ 1052.3125 disk.sda2.out‚ 1091.7890625) proc.memory.availableMB‚ 1293.30078125®¾“2Ã#- _wandbruntime‚9512  _runtime‚9512!M$ÉÄ#à ' lr-AdamW/pg1‚0.00024332913294367972 ' lr-AdamW/pg2‚0.00024332913294367972 trainer/global_step‚8199 % _timestamp‚1.7885198874354928e+09 _runtime‚9512.905978265 _step‚670¿[@sÎÅ#È % _timestamp‚1.7885198874354928e+09 _runtime‚9512.905978265 _step‚670 ' lr-AdamW/pg1‚0.00024332913294367972 ' lr-AdamW/pg2‚0.00024332913294367972 trainer/global_step‚8199ž‡7L2Æ#- _wandbruntime‚9512  _runtime‚9512’Z,þ'Ç#øO QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 205539918 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 168948582 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚618 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 374462412 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 222568448 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177047040 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚682556458637824 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 374488502 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚763120673956864 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9277054284977 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚658271442927952 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407872 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚673243769610752 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚49100262 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚753397809717248 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚682556458637824 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411420 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚753397833657344 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚667548480173569 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚658271425888592 _step‚671 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9722617731072 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9311771616256 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 105057573 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 154157835 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 246508544 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9722840299520 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450780 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007241 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9311771616256 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955591168 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚673243787613696 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 168948584 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9277951696397 trainer/global_step‚8199 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410816 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214665 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚673243787613696 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407872 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 168948584 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 399863808 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚49100878 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 205513830 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚763120427448320 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚667549394624349 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9312671024128 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26088 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 205513830 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚682555541227008 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984´9x…(È#ýO ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚673243787613696 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407872 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 168948584 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 399863808 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚49100878 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 205513830 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚763120427448320 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚667549394624349 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9312671024128 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26088 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 205513830 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚682555541227008 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 205539918 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 168948582 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚618 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 374462412 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 222568448 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177047040 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚682556458637824 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 374488502 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚763120673956864 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9277054284977 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚658271442927952 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407872 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚673243769610752 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚49100262 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚753397809717248 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚682556458637824 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411420 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚753397833657344 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚667548480173569 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚658271425888592 _step‚671 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9722617731072 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9311771616256 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 105057573 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 154157835 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 246508544 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9722840299520 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450780 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007241 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9311771616256 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955591168 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚673243787613696 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 168948584 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9277951696397 trainer/global_step‚8199 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410816 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214665 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984Ÿ²øå2É#- _wandbruntime‚9513  _runtime‚9513ÕøÊôµÊ#€N WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31686 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚606 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1187 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9723800597504 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 168968480 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 205538421 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚49105965 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 399863808 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚658349405982720 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9279060357923 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 205564535 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 105069951 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚763210929535488 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚658349423022080 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007241 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 168968482 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26114 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 374533017 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚763211165544960 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450988 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899421184 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 205538421 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚673323417909248 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899421184 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9312884419584 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9312884419584 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 374506901 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 168968480 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 154175914 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177047040 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917424128 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 205564535 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955591168 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚673323399906304 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 _step‚672 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚608 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 168968482 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411628:Wœ 'Ë#…N KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 374506901 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9313783840768 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 374533017 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚763211165544960 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450988 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899421184 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 205538421 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚673323417909248 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899421184 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9312884419584 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9312884419584 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 374506901 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 168968480 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 154175914 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177047040 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917424128 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 205564535 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955591168 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚673323399906304 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 _step‚672 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚608 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 168968482 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411628 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31686 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚606 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1187 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9723800597504 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 168968480 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 205538421 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚49105965 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 399863808 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚658349405982720 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9279060357923 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 205564535 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 105069951 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚763210929535488 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚658349423022080 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007241 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 168968482 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26114 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9333398353505 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26088 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1187 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411420 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 169943482 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 246508544 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 169943484 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9368325604352 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚49390364 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26090 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚686592710584832 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚671502931537077 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚677225284388352 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚624 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955591168 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 376716304 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_p[…ýÕool.current‚2 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26088 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚662170430594992 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚677225284388352 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407872 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 206746732 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚677225302391296 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚662170447634352 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚686593627995648 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407872 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177047040 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚49389742 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚686593627995648 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 105676375 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 $ _timestamp‚1.788519945058381e+09 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚686592710584832 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450780 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9367426196480 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 399863808 trainer/global_step‚8249 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007241 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 155066739 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564Š3(Ü#üO WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9367426196480 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 399863808 trainer/global_step‚8249 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007241 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 155066739 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9333398353505 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26088 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1187 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411420 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 169943482 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 246508544 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 169943484 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9368325604352 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚49390364 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26090 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚686592710584832 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚671502931537077 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚677225284388352 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚624 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955591168 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 376716304 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26088 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚662170430594992 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚677225284388352 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407872 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 206746732 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚677225302391296 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚662170447634352 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚686593627995648 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407872 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177047040 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚49389742 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚686593627995648 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 105676375 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 $ _timestamp‚1.788519945058381e+09 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚686592710584832 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450780£ O©2Ý#- _wandbruntime‚9571  _runtime‚9571”ŠÏ'Þ#N OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 169963382 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1187 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚686673453683712 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚671582934743511 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26114 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 155084946 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚662248427728480 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450988 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9333609603403 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 169963382 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚767735270101504 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚662248410689120 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 169963380 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 155084314 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9782404779008 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177047040 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007241 _runtime‚9571.668336844 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚686674371107840 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚686674371107840 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177047040 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26114 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 105688749 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899421184 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 trainer/global_step‚8249 _step‚676 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 376734703 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 105688751 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚757953113914880 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚677304914683904 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 376760819 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171lφÍß#†N D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450988 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9333609603403 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 169963382 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚767735270101504 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚662248410689120 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 169963380 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 155084314 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9782404779008 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177047040 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007241 _runtime‚9571.668336844 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚686674371107840 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚686674371107840 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177047040 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26114 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 105688749 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899421184 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 trainer/global_step‚8249 _step‚676 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 376734703 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 105688751 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚757953113914880 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚677304914683904 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 376760819 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 169963382 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1187 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚686673453683712 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚671582934743511 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26114 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 155084946 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚662248427728480¤äÙî¥2à#- _wandbruntime‚9571  _runtime‚9571@äp\á#Ö 0loss/FRt4/assignment_loss‚3.0175449059855555 /loss/FRt2/detection_loss‚1.4294911081031445 /loss/FRt4/detection_loss‚2.8541769212292087 /loss/FRt3/assignment_loss‚2.584189766397079 2loss/FBt1/assignment_loss‚0.007250638765981421 1loss/FBt2/assignment_loss‚0.13454424824562258 1loss/FBt4/assignment_loss‚0.12041651511390228 0loss/FRt1/assignment_loss‚2.0761178994880005 /loss/FRt3/detection_loss‚2.1048134298374257 epoch‚6 0loss/SRqqt4/assignment_loss‚1.46159737173355 /loss/FRt1/detection_loss‚2.4128228791201245 1loss/SRqqt2/detection_loss‚0.7950814277414353 1loss/SRqqt3/detection_loss‚1.6849810847518396 /loss/FBt3/detection_loss‚2.4805978894233705 $loss/total_loss‚37.6170343449907 _step‚677 2loss/SRqqt1/assignment_loss‚1.2097795501826925 1loss/FBt3/assignment_loss‚0.17352721381466835 .loss/FBt1/detection_loss‚2.680327680375841 trainer/global_step‚8249 _runtime‚ 9571.66957993 2loss/SRqqt3/assignment_loss‚0.9368047047102329 1loss/SRqqt1/detection_loss‚1.7653154991963442 0loss/FRt2/assignment_loss‚1.7535922363881142 % _timestamp‚1.7885199461993346e+09 2loss/SRqqt2/assignment_loss‚0.9483749291947817 /loss/SRqqt4/detection_loss‚2.28874769987482 /loss/FBt2/detection_loss‚0.5288303515683579 /loss/FBt4/detection_loss‚2.1681083937486014¬€Ðaâ#Û 2loss/SRqqt3/assignment_loss‚0.9368047047102329 1loss/SRqqt1/detection_loss‚1.7653154991963442 0loss/FRt2/assignment_loss‚1.7535922363881142 % _timestamp‚1.7885199461993346e+09 2loss/SRqqt2/assignment_loss‚0.9483749291947817 /loss/SRqqt4/detection_loss‚2.28874769987482 /loss/FBt2/detection_loss‚0.5288303515683579 /loss/FBt4/detection_loss‚2.1681083937486014 0loss/FRt4/assignment_loss‚3.0175449059855555 /loss/FRt2/detection_loss‚1.4294911081031445 /loss/FRt4/detection_loss‚2.8541769212292087 /loss/FRt3/assignment_loss‚2.584189766397079 2loss/FBt1/assignment_loss‚0.007250638765981421 1loss/FBt2/assignment_loss‚0.13454424824562258 1loss/FBt4/assignment_loss‚0.12041651511390228 0loss/FRt1/assignment_loss‚2.0761178994880005 /loss/FRt3/detection_loss‚2.1048134298374257 epoch‚6 0loss/SRqqt4/assignment_loss‚1.46159737173355 /loss/FRt1/detection_loss‚2.4128228791201245 1loss/SRqqt2/detection_loss‚0.7950814277414353 1loss/SRqqt3/detection_loss‚1.6849810847518396 /loss/FBt3/detection_loss‚2.4805978894233705 $loss/total_loss‚37.6170343449907 _step‚677 2loss/SRqqt1/assignment_loss‚1.2097795501826925 1loss/FBt3/assignment_loss‚0.17352721381466835 .loss/FBt1/detection_loss‚2.680327680375841 trainer/global_step‚8249 _runtime‚ 9571.66957993¥&º=]’ã#‚(:‡ —ÌêÔäºðÿ gpu.0.gpu‚56.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚39.0 gpu.0.powerWatts‚130.232' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚32.558 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0·‹qä#‚(:æ —ÌêÔ—®æ€ disk.md0p1.out‚ 1060.26171875! disk.sda.out‚1100.06103515625 disk.nvme1n1.in‚ 498.80078125$ disk./.usageGB‚2712.442337036133 disk.sdb1.out‚0 disk.md0.in‚ 9.26953125 disk.rbd2.in‚0! disk.sdb.out‚1100.06103515625 disk.rbd0.in‚0 disk.rbd0.out‚ 2.74609375 network.recv‚ 3.074183e+06 disk.sda1.in‚0 disk.dm-0.out‚105252.5 disk.sdb.in‚3.1875 disk.sda.in‚ 6.08203125 disk.sda2.in‚ 6.08203125" disk.sda2.out‚1100.06103515625$ memory_percent‚92.10381507873535 disk.dm-0.in‚ 1116.9140625 disk.md0.out‚ 1060.26171875" disk.sdb2.out‚1100.06103515625 proc.cpu.threads‚17 network.sent‚ 2.1130955e+07) disk./.usagePercent‚75.84102298884315# disk.nvme0n1.out‚54703.15234375 disk.sdb1.in‚0# proc.memory.rssMB‚ 3000.48046875) proc.memory.percent‚18.31347942352295 cpu‚25.196093648582085 disk.sda1.out‚0 disk.md0p1.in‚ 9.26953125 disk.nvme0n1.in‚ 618.11328125# disk.nvme1n1.out‚50549.34765625 disk.rbd2.out‚ 2.74609375 disk.sdb2.in‚3.1875( proc.memory.availableMB‚ 1293.7109375j^e“å#‚(:ˆ ¦ÌêÔ€Îùý gpu.0.gpu‚46.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚169.942' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.4855 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0À +ïqæ#‚(:æ ¦ÌêÔùÇàþ% disk./.usageGB‚2712.5666580200195 disk.md0.out‚ 1062.08984375 disk.dm-0.in‚ 1116.9140625 disk.rbd0.in‚0 disk.rbd0.out‚ 2.74609375 disk.sdb1.in‚0 disk.md0p1.out‚ 1062.08984375 disk.sdb.out‚1101.947265625) proc.memory.percent‚18.31347942352295) disk./.usagePercent‚75.84449905558498 disk.nvme1n1.in‚ 498.80078125 disk.sdb1.out‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.md0p1.in‚ 9.26953125 network.recv‚ 3.080137e+06 disk.sdb2.in‚3.1875# disk.nvme0n1.out‚54791.56640625 disk.sda.in‚ 6.08203125 disk.sda.out‚1101.947265625# proc.memory.rssMB‚ 3000.48046875 cpu‚25.195799254778873# disk.nvme1n1.out‚50629.59765625 disk.rbd2.out‚ 2.74609375 disk.nvme0n1.in‚ 618.11328125 disk.sdb.in‚3.1875$ memory_percent‚92.10841655731201) proc.memory.availableMB‚ 1292.95703125 network.sent‚ 2.1226324e+07 disk.rbd2.in‚0 proc.cpu.threads‚17 disk.sdb2.out‚1101.947265625 disk.md0.in‚ 9.26953125 disk.dm-0.out‚105421.1640625 disk.sda2.in‚ 6.08203125 disk.sda2.out‚1101.947265625Ï “ç#‚(:ˆ µÌêÔ¤Ùþ gpu.0.gpu‚42.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚156.578' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚39.1445 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0’\jè#‚(:ß µÌêÔ½„¥ÿ network.sent‚ 2.1231412e+07) disk./.usagePercent‚75.84449756233754 disk.nvme1n1.in‚ 498.80078125 disk.sda1.out‚0 disk.sdb1.out‚0 disk.sdb2.in‚3.1875 disk.dm-0.out‚ 105449.140625 disk.sda.in‚ 6.08203125 disk.md0.in‚ 9.26953125 disk.dm-0.in‚ 1116.9140625$ memory_percent‚92.10689067840576 disk.sdb.out‚1103.521484375 disk.sda2.in‚ 6.08203125 disk.rbd0.out‚ 2.74609375 disk.sdb1.in‚0 disk.sdb2.out‚1103.521484375 proc.cpu.threads‚17$ disk./.usageGB‚2712.566604614258 disk.sdb.in‚3.1875 disk.md0.out‚ 1063.6015625 disk.md0p1.in‚ 9.26953125 network.recv‚ 3.08305e+06 disk.sda.out‚1103.521484375 disk.sda1.in‚0) proc.memory.availableMB‚ 1293.20703125) proc.memory.percent‚18.31347942352295 disk.sda2.out‚1103.521484375 disk.rbd2.out‚ 2.74609375 disk.nvme0n1.in‚ 618.11328125" disk.nvme0n1.out‚ 54791.5859375# proc.memory.rssMB‚ 3000.48046875 cpu‚25.195543546070628 disk.md0p1.out‚ 1063.6015625 disk.rbd0.in‚0" disk.nvme1n1.out‚ 50657.5546875 disk.rbd2.in‚0àç›v2é#- _wandbruntime‚9622  _runtime‚9622X9Èê# _step‚678 ' lr-AdamW/pg1‚0.00023224467776602944 ' lr-AdamW/pg2‚0.00023224467776602944 trainer/global_step‚8299 % _timestamp‚1.7885199973929818e+09 _runtime‚ 9622.86329356Ð0&Íë#Ç _runtime‚ 9622.86329356 _step‚678 ' lr-AdamW/pg1‚0.00023224467776602944 ' lr-AdamW/pg2‚0.00023224467776602944 trainer/global_step‚8299 % _timestamp‚1.7885199973929818e+09¦|~0©2ì#- _wandbruntime‚9622  _runtime‚9622­o«Sþ'í#øO QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 170938384 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 207979634 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚622 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 106295175 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 216276992 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9388845010613 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 378918016 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 208005722 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407872 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 378944106 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚762329731790848 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26088 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚681206799165952 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410816 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955591168 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚666069452340752 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 207979634 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955591168 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9387947599193 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚620 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 240217088 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26090 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9839554253312 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚690630797353472 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚681206799165952 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 170938382 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚681206817168896 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚690629879942656 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9839770530304 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26088 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚49678962 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 155974757 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1187 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007241 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 106295173 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚49679582 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚772169526261248 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 399863808 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 _step‚679 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚690630797353472 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚681206817168896 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚690629879942656 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411420 trainer/global_step‚8299 _runtime‚9622.864352196 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚772169286044160 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚762329755730944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214665 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚675458297351365 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚666069435301392 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7885199973936996e+09 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9423980184576 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9423080776704 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917410816 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 155974135 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177047040 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407872 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9423980184576 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416Šàµ@(î#ýO N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917410816 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 155974135 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177047040 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407872 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9423980184576 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 170938384 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 207979634 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚622 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 106295175 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 216276992 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9388845010613 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 378918016 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 208005722 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407872 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 378944106 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚762329731790848 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26088 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚681206799165952 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410816 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955591168 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚666069452340752 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 207979634 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955591168 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9387947599193 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚620 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 240217088 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26090 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9839554253312 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚690630797353472 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚681206799165952 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 170938382 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚681206817168896 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚690629879942656 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9839770530304 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26088 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚49678962 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 155974757 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1187 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007241 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 106295173 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚49679582 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚772169526261248 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 399863808 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 _step‚679 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚690630797353472 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚681206817168896 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚690629879942656 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411420 trainer/global_step‚8299 _runtime‚9622.864352196 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚772169286044160 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚762329755730944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214665 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚675458297351365 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚666069435301392 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7885199973936996e+09 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9423980184576 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9423080776704 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090§ØÉñ2ï#- _wandbruntime‚9623  _runtime‚9623E!6$ð#‚N L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 155992183 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9389953672139 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 378988621 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 208030339 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚681286447464448 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9425093001216 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 208004225 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 trainer/global_step‚8299 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177047040 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _runtime‚9623.832792022 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pok¯ËWä ol.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 246495232 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚675536471656031 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚690711540465664 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31686 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899421184 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 170958280 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 155992815 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 208004225 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177047040 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0ã€8U 'ñ#‡N K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955591168 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 399863808 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007241 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 208030339 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 170958282 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 222555136 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899421184 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9389056260511 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9424193580032 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9840942211584 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚681286429461504 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚681286447464448 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚772260017163264 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚762419051011584 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411628 % _timestamp‚1.7885199983616245e+09 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 170958282 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚630 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1187 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚666147415395520 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 208004225 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177047040 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 155992183 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9389953672139 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 378988621 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 208030339 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚681286447464448 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9425093001216 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 208004225 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 trainer/global_step‚8299 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177047040 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _runtime‚9623.832792022 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 246495232 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚675536471656031 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚690711540465664 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31686 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899421184 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 170958280 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 155992815 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 PDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚694667049300480 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9478735356928 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9443394256301 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26088 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 trainer/global_step‚8349 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 209212536 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚685188331946496 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚607 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚776693613896704 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚766795716767744 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917410816 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955591168 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 106913975 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007241 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚605 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚694667049300480 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 171933282 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1212 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26090 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177047040 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 381145818 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 222568448 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _runtime‚9675.005437713 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410816 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9897921069056 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 246508544 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9444291667721 _step‚683 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚685188313943552 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407872 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 209238624 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚49968115 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 171933282 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9479634764800 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407872 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 381171908 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 156882695 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 106913973 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚49968720 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450780 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955591168 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚685H%¼¿I188331946496 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214665 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚694667966711296 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26088 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 399863808 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9479634764800 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 209238624 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 209212536 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚669968457047152 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411420 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 $ _timestamp‚1.788520049534834e+09 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 171933284 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 156882088 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9898143637504 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚679411834264093 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 209238624 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 209212536 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚669968457047152 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411420 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 $ _timestamp‚1.788520049534834e+09 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 171933284 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 156882088 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9898143637504 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚679411834264093 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚694667049300480 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9478735356928 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9443394256301 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26088 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 trainer/global_step‚8349 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 209212536 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚685188331946496 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚607 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚776693613896704 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚766795716767744 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917410816 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955591168 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 106913975 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007241 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚605 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚694667049300480 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 171933282 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1212 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26090 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177047040 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 381145818 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564«ÉšýÐt$‚(:é ñÌêÔÞ€Öÿ disk.md0.in‚ 9.26953125) proc.memory.percent‚18.31347942352295 disk.sda2.in‚ 6.08203125 disk.rbd0.out‚ 2.74609375 disk.sdb2.in‚3.1875# proc.memory.rssMB‚ 3000.48046875 proc.cpu.threads‚17 network.sent‚ 2.1358666e+07" disk.sda2.out‚1110.76513671875 disk.sda1.in‚0 disk.sda1.out‚0 disk.nvme1n1.in‚ 498.80078125 disk.nvme1n1.out‚ 51125.84375" disk.sdb2.out‚1110.76513671875$ memory_percent‚92.11034774780273 network.recv‚ 3.101777e+06% disk./.usageGB‚2712.8698692321777 disk.md0p1.in‚ 9.26953125 disk.rbd0.in‚0' proc.memory.availableMB‚ 1292.640625 cpu‚25.19451238759523 disk.rbd2.out‚ 2.74609375 disk.sdb.in‚3.1875! disk.sdb.out‚1110.76513671875! disk.sda.out‚1110.76513671875) disk./.usagePercent‚75.85297696798068 disk.md0p1.out‚ 1070.59765625# disk.nvme0n1.out‚54936.99609375 disk.rbd2.in‚0 disk.dm-0.in‚ 1116.9140625! disk.dm-0.out‚106062.83984375 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.md0.out‚ 1070.59765625 disk.sda.in‚ 6.08203125 disk.nvme0n1.in‚ 618.11328125˵XO’‚$‚(:‡ ñÌêÔ¬€èÿ gpu.0.gpu‚56.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚37.0 gpu.0.powerWatts‚130.232' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚32.558 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0QѾÈ2ƒ$- _wandbruntime‚9675  _runtime‚9675·k='„$‚N WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚694748709823488 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 171953182 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26114 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 209263241 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 381190307 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 156900245 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9479848160256 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚670046420101920 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 381216423 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚685267962242048 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 106926351 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚670046437141280 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 171953180 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450988 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 218360832 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 _step‚684 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 % _timestamp‚1.7885200505040708e+09 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26114 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚685267944239104 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917424128 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899421184 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 171953182 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚694748709823488 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9899314903040 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 _runtime‚9675.974918338 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 381190307 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 156900851 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9445400329247 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 trainer/global_step‚8349 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚766885035988480 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 106926349 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1187 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411628 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 209263241 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9480747581440 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚49973896 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9444502917619 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 381216423 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31686 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚776784350891520 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899421184 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 171953182 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚694748709823488 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9899314903040 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 _runtime‚9675.974918338 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 381190307 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 156900851 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9445400329247 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 trainer/global_step‚8349 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚766885035988480 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 106926349 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1187 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411628 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 209263241 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9480747581440 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚49973896 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9444502917619 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 381216423 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31686 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚776784350891520 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 381190307 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 156900245 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9479848160256 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚670046420101920 H:DeviceStatsMonitor.on_train_batch_en$a·É d/active.all.allocated‚ 381216423 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚685267962242048 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 106926351 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚670046437141280 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 171953180 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450988 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 218360832 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 _step‚684 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 % _timestamp‚1.7885200505040708e+09 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26114 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚685267944239104 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917424128 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 FDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚698704218658304 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26088 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚634 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 230957056 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 383373620 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407872 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 172928184 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 210445438 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 _step‚687 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407872 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26088 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9534389937152 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚673867444714192 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917410816 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚689169828721152 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 254897152 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 399863808 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚698705136069120 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9956255322624 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 210471526 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚689169828721152 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚50256400 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚698704218658304 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26088 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚634 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 230957056 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 383373620 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407872 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 172928184 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 210445438 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 _step‚687 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407872 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26088 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9534389937152 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚673867444714192 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917410816 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚689169828721152 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 254897152 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 399863808 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚698705136069120 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9956255322624 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 210471526 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚689169828721152 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚50256400 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚689249477019648 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9535502740480 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚673945441847680 TDDeviceStatsMonitor.on_train_batch_endS¡Ìq"/reserved_bytes.large_pool.peak‚ 13037993984 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917424128 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26114 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450988 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007241 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 _runtime‚9731.260816379 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚8399 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899421184 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 172948082 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 _step‚688 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9500846986355 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177047040 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚50262727 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚781308389608448 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9957416523264 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚689249459016704 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899421184 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31686 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 157807264 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚50262115 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚673945424808320 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 172948082 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚683446288834035 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 246495232 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 383444225 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 383444225 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚689249459016704 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 107545149 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚771350973085184 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411628 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 210496143 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955591168 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚781308636103680 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9535502740480 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚614 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9536402161664 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 383418109 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 210496143 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955591168 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚781308636103680 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9535502740480 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚614 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9536402161664 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 383418109 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 210496143 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚689249477019648 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9535502740480 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚673945441847680 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917424128 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26114 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450988 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007241 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 _runtime‚9731.260816379 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚8399 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899421184 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 172948082 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 _step‚688 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9500846986355 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177047040 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚50262727 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚781308389608448 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9957416523264 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚689249459016704 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899421184 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31686 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 157807264 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚50262115 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚673945424808320 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 172948082 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚683446288834035 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 246495232 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 383444225 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 383444225 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚689249459016704 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 107545149 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚771350973085184 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411628 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 210496143 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0°N÷€’2˜$- _wandbruntime‚9731  _runtime‚9731ˆô©`[™$Õ /loss/FRt4/assignment_loss‚2.614649840763637 2loss/SRqqt2/assignment_loss‚1.0354154142622765 /loss/FRt2/detection_loss‚1.7932186925594331 epoch‚6 _step‚689 trainer/global_step‚8399 $ _timestamp‚1.788520105791899e+09 /loss/FRt1/assignment_loss‚2.167409721923911 0loss/FRt3/assignment_loss‚2.6382605204215417 1loss/FBt1/assignment_loss‚0.18459805221452066 1loss/SRqqt4/detection_loss‚2.4418516823878655 ,loss/FBt3/detection_loss‚2.7235392672675 1loss/FBt2/assignment_loss‚0.07891985886679945 /loss/FRt1/detection_loss‚2.0480422867295087 0loss/FBt3/assignment_loss‚0.3882524698939441 /loss/FBt1/detection_loss‚2.5962194030483565 0loss/FRt2/assignment_loss‚1.6926474066103918 2loss/SRqqt1/assignment_loss‚1.2215404475205822 2loss/SRqqt3/assignment_loss‚1.2898491612825267 .loss/FRt3/detection_loss‚2.378133838184369 /loss/FBt4/detection_loss‚2.4227125313546924 &loss/total_loss‚39.415616468582684 _runtime‚9731.262228316 2loss/SRqqt4/assignment_loss‚1.6503216417936177 /loss/FRt4/detection_loss‚2.9283303384269987 1loss/SRqqt2/detection_loss‚0.8487625659658359 0loss/FBt4/assignment_loss‚0.2860076974838093 1loss/SRqqt1/detection_loss‚1.6101641314136876 1loss/SRqqt3/detection_loss‚1.7104112658373736 /loss/FBt2/detection_loss‚0.6663582323694994 Ç`š$Ú 1loss/FBt1/assignment_loss‚0.18459805221452066 1loss/SRqqt4/detection_loss‚2.4418516823878655 ,loss/FBt3/detection_loss‚2.7235392672675 1loss/FBt2/assignment_loss‚0.07891985886679945 /loss/FRt1/detection_loss‚2.0480422867295087 0loss/FBt3/assignment_loss‚0.3882524698939441 /loss/FBt1/detection_loss‚2.5962194030483565 0loss/FRt2/assignment_loss‚1.6926474066103918 2loss/SRqqt1/assignment_loss‚1.2215404475205822 2loss/SRqqt3/assignment_loss‚1.2898491612825267 .loss/FRt3/detection_loss‚2.378133838184369 /loss/FBt4/detection_loss‚2.4227125313546924 &loss/total_loss‚39.415616468582684 _runtime‚9731.262228316 2loss/SRqqt4/assignment_loss‚1.6503216417936177 /loss/FRt4/detection_loss‚2.9283303384269987 1loss/SRqqt2/detection_loss‚0.8487625659658359 0loss/FBt4/assignment_loss‚0.2860076974838093 1loss/SRqqt1/detection_loss‚1.6101641314136876 1loss/SRqqt3/detection_loss‚1.7104112658373736 /loss/FBt2/detection_loss‚0.6663582323694994 /loss/FRt4/assignment_loss‚2.614649840763637 2loss/SRqqt2/assignment_loss‚1.0354154142622765 /loss/FRt2/detection_loss‚1.7932186925594331 epoch‚6 _step‚689 trainer/global_step‚8399 $ _timestamp‚1.788520105791899e+09 /loss/FRt1/assignment_loss‚2.167409721923911 0loss/FRt3/assignment_loss‚2.6382605204215417±Š#’›$‚(:‡ ­ÍêÔ Ã…€ gpu.0.gpu‚57.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚175.844' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.961 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0\^QÆkœ$‚(:à ­ÍêÔÙ‚Ú€ disk.rbd2.out‚ 2.74609375 disk.nvme1n1.in‚ 498.80078125 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.md0.in‚ 9.26953125 disk.dm-0.out‚106588.5$ memory_percent‚92.10741519927979 disk.sda.in‚ 6.08203125 disk.md0.out‚ 1076.953125 disk.md0p1.in‚ 9.26953125! disk.nvme0n1.out‚ 55400.484375) proc.memory.percent‚18.31347942352295 disk.sdb2.in‚3.1875 disk.sdb.in‚3.1875 disk.rbd0.out‚ 2.74609375" disk.sda2.out‚1117.37255859375% disk./.usageGB‚2712.8674392700195 disk.md0p1.out‚ 1076.953125! disk.sdb.out‚1117.37255859375! disk.sda.out‚1117.37255859375 disk.rbd0.in‚0 network.recv‚ 3.120287e+06) disk./.usagePercent‚75.85290902522172 disk.rbd2.in‚0" disk.sdb2.out‚1117.37255859375 disk.nvme0n1.in‚ 618.11328125 disk.sda2.in‚ 6.08203125# proc.memory.rssMB‚ 3000.48046875 network.sent‚ 2.1481982e+07! disk.nvme1n1.out‚ 51188.015625 disk.sda1.in‚0 disk.sda1.out‚0) proc.memory.availableMB‚ 1293.12109375 cpu‚25.193490328610846 disk.dm-0.in‚ 1116.9140625 proc.cpu.threads‚17ìûÍ‹’$‚(:‡ ¼ÍêÔ‚»ôÿ gpu.0.gpu‚47.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚163.09' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚40.7725 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0/Õ²Tlž$‚(:á ¼ÍêÔ³¹³€ disk.sdb.out‚1118.837890625 disk.sda.out‚1118.837890625 disk.rbd0.in‚0 disk.sdb1.out‚0( proc.memory.availableMB‚ 1292.7890625" disk.nvme1n1.out‚ 51188.8359375 disk.sdb2.out‚1118.837890625 disk.md0p1.in‚ 9.26953125! disk.nvme0n1.out‚ 55500.421875 disk.md0.out‚ 1078.35546875 disk.rbd2.in‚0 disk.sda1.out‚0# proc.memory.rssMB‚ 3000.48046875) proc.memory.percent‚18.31347942352295 cpu‚25.193169516701154$ memory_percent‚92.10944175720215 proc.cpu.threads‚17) disk./.usagePercent‚75.86702554664802 disk.dm-0.out‚106689.2578125 disk.rbd0.out‚ 2.74609375 disk.nvme1n1.in‚ 498.80078125 network.recv‚ 3.126129e+06 disk.dm-0.in‚ 1116.9140625 disk.nvme0n1.in‚ 618.11328125 disk.sda2.out‚1118.837890625 disk.sdb1.in‚0 disk.sdb2.in‚3.1875 disk.sda1.in‚0 disk.sdb.in‚3.1875 disk.sda.in‚ 6.08203125 network.sent‚ 2.1592007e+07$ disk./.usageGB‚2713.372314453125 disk.md0p1.out‚ 1078.35546875 disk.sda2.in‚ 6.08203125 disk.md0.in‚ 9.26953125 disk.rbd2.out‚ 2.74609375œg™“Ÿ$‚(:ˆ ËÍêÔɧÿ gpu.0.gpu‚47.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚37.0 gpu.0.powerWatts‚161.254' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚40.3135 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ªPªs $‚(:è ËÍêÔ´î‚€! disk.dm-0.out‚106874.87109375 disk.md0p1.in‚ 9.26953125 disk.nvme0n1.in‚ 618.11328125# proc.memory.rssMB‚ 3000.48046875 network.recv‚ 3.130471e+06$ disk./.usageGB‚2713.495372772217 disk.rbd0.out‚ 2.74609375 disk.rbd2.out‚ 2.74609375! disk.sdb.out‚1120.49072265625 cpu‚25.192850062676985 disk.sda2.in‚ 6.08203125 disk.nvme1n1.in‚ 498.80078125 disk.sdb1.out‚0 disk.rbd2.in‚0 disk.dm-0.in‚ 1116.9140625! disk.sda.out‚1120.49072265625 disk.sda1.in‚0 disk.md0p1.out‚ 1079.9453125$ memory_percent‚92.10786819458008) disk./.usagePercent‚75.87046630875373! disk.nvme1n1.out‚ 51188.953125" disk.sdb2.out‚1120.49072265625 disk.sda1.out‚0# disk.nvme0n1.out‚55685.91796875 disk.sdb.in‚3.1875 disk.sda.in‚ 6.08203125' proc.memory.availableMB‚ 1293.046875 disk.rbd0.in‚0) proc.memory.percent‚18.31347942352295 network.sent‚ 2.1598017e+07 disk.md0.in‚ 9.26953125 disk.md0.out‚ 1079.9453125 disk.sdb2.in‚3.1875 proc.cpu.threads‚17" disk.sda2.out‚1120.49072265625 disk.sdb1.in‚0ƒmcž¡$‚(:“ ÚÍêÔ›¤‚þ gpu.0.gpu‚49.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚115.634' gpu.0.enforcedPowerLimitWatts‚400.0) gpu.0.powerPercent‚28.908499999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 tCu¢$‚(:ê ÚÍêÔ›²¹ÿ network.recv‚ 3.135083e+06 disk.md0p1.in‚ 9.26953125 disk.rbd0.out‚ 2.74609375# proc.memory.rssMB‚ 3000.48046875 cpu‚25.192532656351844 proc.cpu.threads‚17 disk.sda1.out‚0 disk.sdb.in‚3.1875 disk.sda2.in‚ 6.08203125 disk.rbd2.in‚0" disk.sdb2.out‚1122.52001953125) disk./.usagePercent‚75.87396082776763 disk.md0.in‚ 9.26953125 disk.sda1.in‚0$ memory_percent‚92.10946559906006 disk.nvme0n1.out‚ 56070.15625! disk.sdb.out‚1122.52001953125! disk.sda.out‚1122.52001953125 disk.rbd0.in‚0 disk.sdb1.out‚0! disk.dm-0.out‚107255.06640625) proc.memory.availableMB‚ 1292.78515625 disk.md0p1.out‚ 1081.9140625 disk.nvme0n1.in‚ 618.11328125 disk.md0.out‚ 1081.9140625) proc.memory.percent‚18.31347942352295 disk.sda.in‚ 6.08203125 disk.sdb1.in‚0 disk.rbd2.out‚ 2.74609375 disk.sdb2.in‚3.1875 network.sent‚ 2.1604299e+07% disk./.usageGB‚2713.6203536987305 disk.nvme1n1.in‚ 498.80078125# disk.nvme1n1.out‚51189.16015625 disk.dm-0.in‚ 1116.9140625" disk.sda2.out‚1122.52001953125<ƒW%2£$- _wandbruntime‚9789  _runtime‚9789¸Çɤ$à _runtime‚9789.892868751 _step‚690 ' lr-AdamW/pg1‚0.00021592812155281122 ' lr-AdamW/pg2‚0.00021592812155281122 trainer/global_step‚8449 % _timestamp‚1.7885201644224935e+09®9šøÎ¥$È _step‚690 ' lr-AdamW/pg1‚0.00021592812155281122 ' lr-AdamW/pg2‚0.00021592812155281122 trainer/global_step‚8449 % _timestamp‚1.7885201644224935e+09 _runtime‚9789.892868751² üú2¦$- _wandbruntime‚9789  _runtime‚9789§$úO TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917410816 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 158697692 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚785742270502912 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚50546119 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 246508544 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407872 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411420 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9554287570517 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚687321651441889 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214665 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚693151343498752 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 173923084 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177047040 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚603 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 211678340 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 108151575 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚677766449420592 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450780 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9590044517376 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚702741388016128 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 173923084 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚693151361501696 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1187 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚677766466459952 _step‚691 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚702742305426944 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9590943925248 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚785742517011456 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 173923082 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 211704428 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚693151361501696 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10014878170112 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _runtime‚9789.893959046 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 385601422 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚702741388016128 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 385627512 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 173923082 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955591168 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚775727638841344 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007241 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 % _timestamp‚1.7885201644235263e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 108151573 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9555184981937 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 211704428 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚605 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26090 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407872í³æ¯(¨$ÿO H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 173923082 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 211704428 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚693151361501696 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10014878170112 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _runtime‚9789.893959046 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 385601422 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚702741388016128 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 385627512 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 173923082 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955591168 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚775727638841344 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007241 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 % _timestamp‚1.7885201644235263e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 108151573 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9555184981937 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 211704428 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚605 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26090 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407872 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917410816 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 158697692 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚785742270502912 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚50546119 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 246508544 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407872 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411420 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9554287570517 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚687321651441889 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214665 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚693151343498752 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 173923084 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177047040 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚603 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 211678340 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 108151575 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚677766449420592 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450780 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9590044517376 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚702741388016128 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 173923084 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚693151361501696 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1187 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚677766466459952 _step‚691 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚702742305426944 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9590943925248 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚785742517011456³ÊQÐF2©$- _wandbruntime‚9790  _runtime‚9790!yxl 'ª$ƒN >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 385672027 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 250689536 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10016045026304 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 211729045 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955591168 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9591157320704 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚702823048539136 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚50552450 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚687399825746555 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚693230973794304 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 211702931 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚687400740197543 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917424128 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚614 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10015818276864 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007241 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 158715787 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 108163951 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 173942982 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 108163949 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚775816958062080 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0}€Í¥«$ˆN L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 158715787 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 108163951 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 173942982 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 108163949 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚775816958062080 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 385672027 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 250689536 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10016045026304 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 211729045 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955591168 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9591157320704 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚702823048539136 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚50552450 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚687399825746555 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚693230973794304 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 211702931 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚687400740197543 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚øèP i  917424128 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚614 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10015818276864 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007241 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564´VÈ{™2¬$- _wandbruntime‚9790  _runtime‚9790´D‘V­$Ð 2loss/SRqqt4/assignment_loss‚1.6508677320089191 3loss/FBt1/assignment_loss‚0.0011548983862515645 0loss/FBt2/assignment_loss‚0.1420264920642392 1loss/FBt3/assignment_loss‚0.13231966182568364 0loss/SRqqt4/detection_loss‚2.397703162394464 epoch‚6 2loss/SRqqt3/assignment_loss‚1.2662850648359678 0loss/FBt4/assignment_loss‚0.3415982484148117 /loss/FRt1/detection_loss‚1.8684775367330047 .loss/FRt4/detection_loss‚3.176627677220565 /loss/FBt2/detection_loss‚0.7048659516547463 /loss/FRt1/assignment_loss‚2.826376953085915 .loss/FBt1/detection_loss‚3.314843734105428 0loss/FRt2/assignment_loss‚1.0491337284707187 -loss/FRt4/assignment_loss‚3.1733039525839 2loss/SRqqt1/assignment_loss‚1.2676924516646457 .loss/FRt2/detection_loss‚1.621750448029815 /loss/FBt4/detection_loss‚2.3124470114707947 trainer/global_step‚8449 $ _timestamp‚1.788520165391571e+09 2loss/SRqqt2/assignment_loss‚0.8760099003063233 0loss/SRqqt2/detection_loss‚0.925421564670449 1loss/SRqqt3/detection_loss‚1.7517974801188292 %loss/total_loss‚38.95737980587093 _runtime‚ 9790.86170544 /loss/FRt3/assignment_loss‚2.139223114365623 .loss/FRt3/detection_loss‚2.277313330875976 1loss/SRqqt1/detection_loss‚1.6585887846137797 /loss/FBt3/detection_loss‚2.0815509259700775 _step‚693fPT¤[®$Õ /loss/FRt3/assignment_loss‚2.139223114365623 .loss/FRt3/detection_loss‚2.277313330875976 1loss/SRqqt1/detection_loss‚1.6585887846137797 /loss/FBt3/detection_loss‚2.0815509259700775 _step‚693 2loss/SRqqt4/assignment_loss‚1.6508677320089191 3loss/FBt1/assignment_loss‚0.0011548983862515645 0loss/FBt2/assignment_loss‚0.1420264920642392 1loss/FBt3/assignment_loss‚0.13231966182568364 0loss/SRqqt4/detection_loss‚2.397703162394464 epoch‚6 2loss/SRqqt3/assignment_loss‚1.2662850648359678 0loss/FBt4/assignment_loss‚0.3415982484148117 /loss/FRt1/detection_loss‚1.8684775367330047 .loss/FRt4/detection_loss‚3.176627677220565 /loss/FBt2/detection_loss‚0.7048659516547463 /loss/FRt1/assignment_loss‚2.826376953085915 .loss/FBt1/detection_loss‚3.314843734105428 0loss/FRt2/assignment_loss‚1.0491337284707187 -loss/FRt4/assignment_loss‚3.1733039525839 2loss/SRqqt1/assignment_loss‚1.2676924516646457 .loss/FRt2/detection_loss‚1.621750448029815 /loss/FBt4/detection_loss‚2.3124470114707947 trainer/global_step‚8449 $ _timestamp‚1.788520165391571e+09 2loss/SRqqt2/assignment_loss‚0.8760099003063233 0loss/SRqqt2/detection_loss‚0.925421564670449 1loss/SRqqt3/detection_loss‚1.7517974801188292 %loss/total_loss‚38.95737980587093 _runtime‚ 9790.86170544µ‘¡¯$‚(:… éÍêÔÌËþ gpu.0.gpu‚35.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚157.52' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚39.38 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ðDp¢r°$‚(:ç éÍêÔûÔúÿ network.recv‚ 3.138285e+06 disk.nvme0n1.in‚ 618.11328125 disk.rbd0.in‚0 disk.rbd0.out‚ 2.74609375 disk.md0p1.out‚ 1083.41015625 disk.sdb.in‚3.1875( proc.memory.availableMB‚ 1292.4765625 network.sent‚ 2.1607395e+07 disk.sda2.in‚ 6.08203125 disk.nvme1n1.in‚ 498.80078125% disk./.usageGB‚2713.6203536987305 disk.sdb2.in‚3.1875 disk.dm-0.in‚ 1116.9140625 disk.rbd2.out‚ 2.74609375$ memory_percent‚92.11134910583496 disk.sdb1.in‚0 disk.md0.out‚ 1083.41015625) disk./.usagePercent‚75.87396082776763! disk.sdb2.out‚1124.0791015625 disk.sda1.out‚0! disk.sda2.out‚1124.0791015625 disk.md0p1.in‚ 9.26953125 disk.sda.out‚1124.0791015625! disk.dm-0.out‚107333.65234375 disk.sdb.out‚1124.0791015625 disk.sda.in‚ 6.08203125# disk.nvme1n1.out‚51189.35546875 disk.sdb1.out‚0 disk.md0.in‚ 9.26953125# proc.memory.rssMB‚ 3000.48046875 cpu‚25.19228682382586 disk.sda1.in‚0! disk.nvme0n1.out‚ 56144.296875 disk.rbd2.in‚0) proc.memory.percent‚18.31347942352295 proc.cpu.threads‚17ÍvPr”±$‚(:‰ øÍêÔÔ™Œþ gpu.0.gpu‚60.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚37.0 gpu.0.powerWatts‚116.583' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚29.14575 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¤ù=s²$‚(:è øÍêÔœó©ÿ) proc.memory.availableMB‚ 1292.22265625 disk.dm-0.in‚ 1116.9140625% disk./.usageGB‚2713.8713150024414 disk.rbd0.in‚0$ memory_percent‚92.11289882659912! proc.memory.rssMB‚ 3001.234375 disk.md0.in‚ 9.26953125" disk.sdb2.out‚1125.62451171875 network.recv‚ 3.142825e+06 disk.sda.in‚ 6.08203125 disk.nvme0n1.in‚ 618.11328125 disk.sdb.in‚3.1875! disk.sdb.out‚1125.62451171875 disk.md0p1.in‚ 9.26953125) proc.memory.percent‚18.31808090209961 cpu‚25.193699881564147 disk.sda1.out‚0 network.sent‚ 2.1714248e+07! disk.sda.out‚1125.62451171875" disk.sda2.out‚1125.62451171875 disk.nvme1n1.out‚ 51189.875 disk.sdb1.out‚0 disk.md0.out‚ 1084.89453125# disk.nvme0n1.out‚56144.60546875 disk.sda2.in‚ 6.08203125 disk.rbd0.out‚ 2.74609375 disk.sdb1.in‚0 disk.sda1.in‚0! disk.dm-0.out‚107334.48046875 proc.cpu.threads‚17) disk./.usagePercent‚75.88097781085487 disk.nvme1n1.in‚ 498.80078125 disk.rbd2.in‚0 disk.rbd2.out‚ 2.74609375 disk.sdb2.in‚3.1875 disk.md0p1.out‚ 1084.894531259Îŵ“³$‚(:ˆ ‡ÎêÔꤛÿ gpu.0.gpu‚2.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚37.0 gpu.0.powerWatts‚182.641' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚45.66025 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Šû«˜i´$‚(:Þ ‡ÎêÔйˆ€ disk.sda.in‚ 6.08203125 disk.sda1.out‚0 disk.md0p1.in‚ 9.26953125 proc.cpu.threads‚17 disk.rbd0.in‚0" disk.nvme1n1.out‚ 51190.0390625 disk.md0.in‚ 9.26953125 cpu‚25.198299226439566% disk./.usageGB‚2712.8166732788086" disk.sda2.out‚1127.10693359375 disk.rbd2.out‚ 2.74609375 disk.dm-0.out‚ 107828.796875 disk.nvme0n1.in‚ 618.11328125 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.md0.out‚ 1086.3125) proc.memory.percent‚18.31808090209961 disk.nvme1n1.in‚ 498.80078125 disk.rbd2.in‚0! proc.memory.rssMB‚ 3001.234375 network.recv‚ 3.146955e+06! disk.sda.out‚1127.10693359375 disk.sdb2.in‚3.1875 disk.sdb.in‚3.1875$ memory_percent‚92.11292266845703" disk.nvme0n1.out‚ 56638.7578125 network.sent‚ 2.1722304e+07 disk.sda2.in‚ 6.08203125 disk.dm-0.in‚ 1116.9140625 disk.md0p1.out‚ 1086.3125! disk.sdb.out‚1127.10693359375) disk./.usagePercent‚75.85148958686024 disk.rbd0.out‚ 2.74609375" disk.sdb2.out‚1127.10693359375 disk.sda1.in‚0& proc.memory.availableMB‚ 1292.21875vaçA“µ$‚(:ˆ –ÎêÔ™Þþ gpu.0.gpu‚35.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚36.0 gpu.0.powerWatts‚113.798' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.4495 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ŠþÂŽj¶$‚(:ß –ÎêÔƒ«€ÿ network.sent‚ 2.172648e+07 disk.nvme1n1.in‚ 498.80078125 disk.md0.out‚ 1088.11328125 disk.md0p1.out‚ 1088.11328125% disk./.usageGB‚2712.8165016174316 disk.sdb.in‚3.1875# disk.nvme1n1.out‚51190.21484375 disk.sdb.out‚1128.970703125 disk.sda2.out‚1128.970703125 disk.sda1.out‚0$ memory_percent‚92.11530685424805 disk.sda.in‚ 6.08203125 disk.md0.in‚ 9.26953125 disk.rbd2.in‚0 disk.dm-0.in‚ 1116.9140625! disk.dm-0.out‚107961.30859375 disk.nvme0n1.out‚ 56771.09375! proc.memory.rssMB‚ 3001.234375 proc.cpu.threads‚17 disk.sda1.in‚0 network.recv‚ 3.151991e+06 disk.rbd0.in‚0 disk.sdb1.out‚0 disk.sdb2.in‚3.1875 disk.sdb2.out‚1128.970703125' proc.memory.availableMB‚ 1291.828125) proc.memory.percent‚18.31808090209961 cpu‚25.204845522624513 disk.sda2.in‚ 6.08203125 disk.rbd0.out‚ 2.74609375 disk.rbd2.out‚ 2.74609375 disk.md0p1.in‚ 9.26953125 disk.nvme0n1.in‚ 618.11328125) disk./.usagePercent‚75.85148478713629 disk.sda.out‚1128.970703125 disk.sdb1.in‚0]ß.og·$‚(:Ü ¥ÎêÔÝ”ëÿ disk.md0.in‚ 9.26953125 disk.dm-0.in‚ 1116.9140625 disk.md0p1.out‚ 1089.5625( proc.memory.availableMB‚ 1292.3359375 network.recv‚ 3.155411e+06! disk.sda.out‚1130.48291015625 disk.sdb1.out‚0 disk.sdb.in‚3.1875 disk.sda2.in‚ 6.08203125 disk.nvme1n1.out‚ 51190.375$ memory_percent‚92.11220741271973 cpu‚25.211083473381617 disk.rbd2.in‚0 disk.rbd2.out‚ 2.74609375" disk.sdb2.out‚1130.48291015625 disk.nvme0n1.in‚ 618.11328125) proc.memory.percent‚18.31808090209961 network.sent‚ 2.1731732e+07) disk./.usagePercent‚75.85148340054938 disk.sdb2.in‚3.1875 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-0.out‚108133.0859375! disk.sdb.out‚1130.48291015625" disk.sda2.out‚1130.48291015625 proc.cpu.threads‚17$ disk./.usageGB‚2712.816452026367 disk.rbd0.in‚0 disk.rbd0.out‚ 2.74609375! proc.memory.rssMB‚ 3001.234375 disk.md0.out‚ 1089.5625 disk.md0p1.in‚ 9.26953125" disk.nvme0n1.out‚ 56942.7109375 disk.sda.in‚ 6.08203125 disk.nvme1n1.in‚ 498.80078125 disk.sdb1.in‚0ˆþÓO’¸$‚(:‡ ¥ÎêԦ𭀠gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚35.0 gpu.0.powerWatts‚113.466' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.3665 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‹Väæ2¹$- _wandbruntime‚9862  _runtime‚9862à_3£g*º$áT `PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 _step‚694 OADeviceStatsMonitor.on_validation_batch_start/active.all.allocated‚ 387428343 \LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 ]ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.current‚ 675116544 aTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.current‚23940096 eRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.freed‚10055233587200 \ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.current‚17039360 _MDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.freed‚ 9593909602765 LCDeviceStatsMonitor.on_validation_batch_start/active.large_pool.peak‚3677 VHDeviceStatsMonitor.on_validation_batch_start/active.small_pool.allocated‚ 212478163 PGDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.peak‚3677 QGDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.peak‚31686 J@DeviceStatsMonitor.on_validation_batch_start/allocation.all.peak‚31931 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.freed‚50890837 NHDeviceStatsMonitor.on_validation_batch_start/oversize_segments.allocated‚0 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.freed‚0 OADeviceStatsMonitor.on_validation_batch_start/allocation.all.freed‚ 387407822 OGDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.current‚201 RKDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.peak‚35 OIDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.current‚0 TDDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.peak‚ 14220787712 K=DeviceStatsMonitor.on_validation_batch_start/active.all.freed‚ 387407822 aMDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.freed‚696891160149504 iVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.allocated‚10055378457088 QIDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.allocated‚607 cQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.allocated‚ 9594582747525 RBDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.peak‚ 13955591168 cQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.allocated‚ 9630478504448 KEDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.freed‚0 [GDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.allocated‚706521656656896 ZLDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.allocated‚ 212478163 VHDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.freed‚ 212457644 B;DeviceStatsMonitor.on_validation_batch_start/max_split_size‚-1 _runtime‚9862.561619231 WIDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.allocated‚ 159678435 RDDeviceStatsMonitor.on_validation_batch_start/active.small_pool.freed‚ 212457644 MCDeviceStatsMonitor.on_validation_batch_start/active.small_pool.peak‚31686 `NDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.allocated‚ 9630478504448 [LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.peak‚ 1175007241 I@DeviceStatsMonitor.on_validation_batch_start/segment.all.current‚1171 YIDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.allocated‚ 14220787712 MDDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.peak‚1212 eQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.allocated‚696891178152448 ZLDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.allocated‚ 174950180 bTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.current‚ 144869888 E?DeviceStatsMonitor.on_validation_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.freed‚0 eQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.allocated‚681296922179120 I?DeviceStatsMonitor.on_validation_batch_start/active.all.current‚20521 _MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.freed‚ 9629803387904 SEDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.freed‚ 159678234 UEDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.peak‚ 13510214665 OGDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.current‚607 QIDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.allocated‚564 XJDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.peak‚ 399863808 \LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.peak‚ 12392107008 % _timestamp‚1.7885202370907633e+09 LFDeviceStatsMonitor.on_validation_batch_start/active.large_pool.current‚2 ^JDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.allocated‚706521656656896 _KDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.freed‚789750341706752 VHDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.current‚ 690184120 LDDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.peak‚564 XIDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.peak‚ 1177047040 [MDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.current‚ 168809984 ICDeviceStatsMonitor.on_validation_batch_start/oversize_segments.peak‚0 ^JDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.allocated‚690891504926645 F=DeviceStatsMonitor.on_validation_batch_start/segment.all.peak‚1171 bNDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.allocated‚696891178152448 UEDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.peak‚ 13955591168 SEDeviceStatsMonitor.on_validation_batch_start/allocation.all.allocated‚ 387428343 ZFDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.freed‚690890814742525 WCDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.freed‚706520963537408 ZLDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.current‚ 675116544 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 ;5DeviceStatsMonitor.on_validation_batch_start/num_ooms‚0 ^NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.current‚ 13037993984 VHDeviceStatsMonitor.on_validation_batch_start/active.large_pool.allocated‚ 174950180 LDDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.peak‚607 OGDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.current‚564 PJDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.current‚2 TJDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.current‚20519 jVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.allocated‚779695132059648 [KDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 SEDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.current‚ 693119488 VNDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.current‚199 JDDeviceStatsMonitor.on_validation_batch_start/oversize_segments.freed‚0 ZKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 KBDeviceStatsMonitor.on_validation_batch_start/segment.all.allocated‚1171 trainer/global_step‚8462 YLDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.current‚18002944 cODeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.allocated‚789750510516736 F=DeviceStatsMonitor.on_validation_batch_start/num_device_alloc‚1171 KEDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.freed‚0 \JDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.freed‚ 9629803387904 KEDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.freed‚0 ZFDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.freed‚706520963537408 ^PDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.allocated‚ 108787399 TKDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.peak‚1187 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 QKDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.allocated‚0 WGDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.current‚ 14220787712 _PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 ]NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.current‚ 1182793728 TNDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.current‚2 ^JDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.freed‚696891160149504 VHDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.freed‚ 174950178 D>DeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 RDDeviceStatsMonitor.on_validation_batch_start/active.large_pool.freed‚ 174950178 MCDeviceStatsMonitor.on_validation_batch_start/allocation.all.current‚20521 LFDeviceStatsMonitor.on_validation_batch_start/oversize_segments.current‚0 ]ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.current‚ 673144760 LFDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.peak‚0 PFDeviceStatsMonitor.on_validation_batch_start/active.small_pool.current‚20519 [LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 aMDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.freed‚681296905139760 D>DeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 ZLDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.freed‚ 108787397 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.freed‚0 YIDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.peak‚ 12835443712 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.allocated‚50891036 BDeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 RDDeviceStatsMonitor.on_validation_batch_start/active.large_pool.freed‚ 174950178 MCDeviceStatsMonitor.on_validation_batch_start/allocation.all.current‚20521 LFDeviceStatsMonitor.on_validation_batch_start/oversize_segments.current‚0 ]ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.current‚ 673144760 LFDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.peak‚0 PFDeviceStatsMonitor.on_validation_batch_start/active.small_pool.current‚20519 [LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 aMDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.freed‚681296905139760 D>DeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 ZLDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.freed‚ 108787397 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.freed‚0 YIDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.peak‚ 12835443712 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.allocated‚50891036 BDeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚31931 cPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.freed‚10055722149888 OGDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.allocated‚564 @9DeviceStatsMonitor.on_validation_batch_end/max_split_size‚-1 ZJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 TFDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.freed‚ 212466418 `RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.current‚ 144869888 XJDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.allocated‚ 212486937 ]IDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.freed‚789806279741440 ZJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.peak‚ 12392107008 XDDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.freed‚706575023375872 TFDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.freed‚ 174965454 aMDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.allocated‚789806448551424 XDDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.freed‚690941071759073 ]NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 TFDeviceStatsMonitor.on_validation_batch_end/active.large_pool.allocated‚ 174965456 RHDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.current‚20519 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 D;DeviceStatsMonitor.on_validation_batch_end/num_device_alloc‚1171 BDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚1171 JDDeviceStatsMonitor.on_validation_batch_end/active.large_pool.current‚2 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.freed‚50896122 dPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.freed‚779750557591552 93DeviceStatsMonitor.on_validation_batch_end/num_ooms‚0 PBDeviceStatsMonitor.on_validation_batch_end/active.small_pool.freed‚ 212466418 QCDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.current‚ 693119488 ZHDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.freed‚ 9630291950592 ]KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.freed‚ 9594396529489 MEDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.current‚564 YEDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.allocated‚706575716495360 JDDeviceStatsMonitor.on_validation_batch_end/oversize_segments.current‚0 ICDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7885202381816766e+09 _runtime‚9863.652624822 G=DeviceStatsMonitor.on_validation_batch_end/active.all.current‚20521 PBDeviceStatsMonitor.on_validation_batch_end/active.large_pool.freed‚ 174965454 YJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.peak‚ 1177047040 XJDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.allocated‚ 174965456 YKDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.current‚ 168809984 LFDeviceStatsMonitor.on_validation_batch_end/oversize_segments.allocated‚0 _KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.freed‚681346675229584ŽÀ°kv)¾$ðR KBDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.peak‚1212 @:DeviceStatsMonitor.on_validation_batch_end/num_device_free‚0 TFDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.current‚ 690184120 SCDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.peak‚ 13510214665 ^NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 D;DeviceStatsMonitor.on_validation_batch_end/segment.all.peak‚1171 I;DeviceStatsMonitor.on_validation_batch_end/active.all.freed‚ 387431872 UADeviceStatsMonitor.on_validation_batch_end/active_bytes.all.freed‚706575023375872 NHDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.current‚2 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 I@DeviceStatsMonitor.on_validation_batch_end/segment.all.allocated‚1171 G>DeviceStatsMonitor.on_validation_batch_end/segment.all.current‚1171 JDDeviceStatsMonitor.on_validation_batch_end/active.large_pool.current‚2 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.freed‚50896122 dPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.freed‚779750557591552 93DeviceStatsMonitor.on_validation_batch_end/num_ooms‚0 PBDeviceStatsMonitor.on_validation_batch_end/active.small_pool.freed‚ 212466418 QCDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.current‚ 693119488 ZHDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.freed‚ 9630291950592 ]KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.freed‚ 9594396529489 MEDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.current‚564 YEDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.allocated‚706575716495360 JDDeviceStatsMonitor.on_validation_batch_end/oversize_segments.current‚0 ICDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7885202381816766e+09 _runtime‚9863.652624822 G=DeviceStatsMonitor.on_validation_batch_end/active.all.current‚20521 PBDeviceStatsMonitor.on_validation_batch_end/active.large_pool.freed‚ 174965454 YJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.peak‚ 1177047040 XJDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.allocated‚ 174965456 YKDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.current‚ 168809984 LFDeviceStatsMonitor.on_validation_batch_end/oversize_segments.allocated‚0 _KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.freed‚681346675229584 trainer/global_step‚8462 OEDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.peak‚31686 cODeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.allocated‚681346692268944 M?DeviceStatsMonitor.on_validation_batch_end/active.all.allocated‚ 387452393 YJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.peak‚ 1175007241 _step‚695 \NDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.allocated‚ 108796839 P@DeviceStatsMonitor.on_validation_batch_end/active_bytes.all.peak‚ 13955591168 VGDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.peak‚ 1177047040 \HDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.allocated‚706575716495360 gTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.allocated‚10055867019776 aODeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.allocated‚ 9595069674249 WGDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.allocated‚ 14220787712 [LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.current‚ 1182793728 XJDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.current‚ 675116544 cODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.allocated‚696944749428224 KEDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.peak‚ 12835443712 aODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.allocated‚ 9630967067136 C=DeviceStatsMonitor.on_validation_batch_end/num_oom_rejections‚0 MEDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.current‚607 H>DeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚31931 cPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.freed‚10055722149888 OGDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.allocated‚564 @9DeviceStatsMonitor.on_validation_batch_end/max_split_size‚-1 ZJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 TFDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.freed‚ 212466418 `RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.current‚ 144869888 XJDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.allocated‚ 212486937 ]IDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.freed‚789806279741440 ZJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.peak‚ 12392107008 XDDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.freed‚706575023375872 TFDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.freed‚ 174965454 aMDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.allocated‚789806448551424 XDDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.freed‚690941071759073 ]NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 TFDeviceStatsMonitor.on_validation_batch_end/active.large_pool.allocated‚ 174965456 RHDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.current‚20519 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 D;DeviceStatsMonitor.on_validation_batch_end/num_device_alloc‚1171 B&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.3028632402420044 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.7142857313156128 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 :"Purity/0FRt0SRqqt2FBt/event_purity‚0.36797860264778137 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.2623692452907562 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.30873072147369385 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 jet/accuracy_1_of_6‚1 particle/accuracy_5_of_6‚1 ("Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.4922527074813843 8"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.410216361284256 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9904694557189941 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9942528009414673 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 particle/accuracy_4_of_7‚1 6 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.274710476398468 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 ) Purity/4FRt1SRqqt*FBt/FRt_purity‚0.25 + Purity/1FRt0SRqqt3FBt/FRt_purity‚0.5625 8"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.696969747543335 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9890053868293762 7 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.2739463746547699 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 7 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.3194443881511688 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 & Purity/1FRt0SRqqt4FBt/FRt_purity‚1 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 particle/accuracy_5_of_7‚1 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 jet/accuracy_4_of_6‚0 6 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.949999988079071 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.4642857015132904 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 ;"Purity/*FRt1SRqqt2FBt/event_purity‚0.027118537575006485 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 jet/accuracy_6_of_7‚0 particle/accuracy_6_of_7‚1 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.5366758704185486 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.24767287075519562 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 particle/accuracy_3_of_7‚1 *"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.5 particle/accuracy_7_of_7‚1 8"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.699999988079071 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.3700917065143585 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.09598515182733536 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 +jet/accuracy_1_of_1‚0.43919119238853455 particle/accuracy_1_of_3‚1 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9703981876373291 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 +"Purity/1FRt0SRqqt3FBt/event_purity‚0.25 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.5845587253570557 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 +jet/accuracy_3_of_5‚0.08639272302389145 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 *"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.5 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9969581365585327 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5386456251144409 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.10549060255289078 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5461407899856567 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9351851940155029 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.02865961380302906 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 :"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.48337745666503906 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.21783624589443207 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 particle/accuracy_4_of_5‚1 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.19696970283985138 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.4675925672054291 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 +"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.25 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 5 Purity/0FRt1SRqqt3FBt/FBt_purity‚ê­ÉÐcx0.80952388048172 +"Purity/1FRt1SRqqt3FBt/SRqqt_purity‚0.75 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.24111545085906982 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.2936336100101471 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 (particle/accuracy‚0.8564813137054443 7 Purity/*FRt3SRqqt1FBt/FBt_purity‚0.9629629850387573 particle/accuracy_4_of_6‚1 jet/accuracy_2_of_7‚0.5 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.4000000059604645 ;"Purity/*FRt3SRqqt0FBt/event_purity‚0.008064515888690948 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9680315852165222 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 ) Purity/*FRt1SRqqt3FBt/FRt_purity‚0.75 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.11004329472780228 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚1 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.3915492594242096 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9909475445747375 ;"Purity/0FRt3SRqqt*FBt/event_purity‚0.026143791154026985 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 ) Purity/3FRt3SRqqt0FBt/FRt_purity‚0.25 +"Purity/1FRt*SRqqt3FBt/SRqqt_purity‚0.75 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.19287176430225372 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 particle/accuracy_4_of_4‚1 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.28339147567749023 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.11403507739305496 9"Purity/0FRt0SRqqt3FBt/event_purity‚0.1354166567325592 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.3472222089767456 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 :"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.37006619572639465 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.18281467258930206 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.24881884455680847 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.25174012780189514 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 9"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.3840160667896271 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.5780249238014221 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 7 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.2666666805744171 particle/accuracy_6_of_6‚1 9"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.3561002314090729 7 Purity/0FRt3SRqqt1FBt/FBt_purity‚0.8888888955116272 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.1865079402923584 9"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.3912280797958374 :"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.45690351724624634 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.5556091070175171 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.25938597321510315 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.26929011940956116 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9969576597213745 particle/accuracy_1_of_6‚1 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.3571428656578064 particle/accuracy_1_of_7‚1 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9904512166976929 8 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.35273367166519165 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 8"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.439007043838501 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.24172650277614594 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 *jet/accuracy_3_of_6‚0.1538461595773697 ("Purity/4FRt*SRqqt*FBt/SRqqt_purity‚1 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 8 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.40776312351226807 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 7 Purity/1FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 6 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.932539701461792 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 <"Purity/1FRt2SRqqt*FBt/event_purity‚0.0025012025143951178 *jet/accuracy_1_of_2‚0.7292569279670715 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 ( Purity/1FRt3SRqqt2FBt/FBt_purity‚0.5 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 particle/accuracy_2_of_7‚1 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 6 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.989056408405304 *loss/val_total_loss‚38.623757255404314 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.5206896066665649 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 * Purity/0FRt0SRqqt4FBt/FBt_purity‚0.875 jet/accuracy_6_of_6‚0 :"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.35185182094573975 :"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.41118374466896057 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.3645538091659546 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.023960694670677185 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 +jet/accuracy_2_of_2‚0.10485740751028061 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.1944444626569748 & Purity/*FRt3SRqqt2FBt/FRt_purity‚1 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.14138785004615784 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 7 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.2380952537059784 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6829110383987427 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 7 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.2593677043914795 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 jet/accuracy_5_of_5‚0 particle/accuracy_2_of_4‚1 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.3402777910232544 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 jet/accuracy_5_of_6‚0 7 Purity/3FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 jet/accuracy_3_of_7‚0 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 % _timestamp‚1.7885202528992724e+09 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9739582538604736 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.25718387961387634 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.5028735995292664 +particle/specificity‚0.9511528611183167 'particle/f_score‚0.4285372197628021 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚1 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.20000000298023224 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9921626448631287 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 :"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.44581881165504456 :"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.43313494324684143 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 particle/accuracy_1_of_4‚1 9"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.5066666603088379 ("Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 :"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.42717739939689636 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.23731723427772522 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 6 Purity/1FRt3SRqqt*FBt/FBt_purity‚0.970588207244873 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 particle/accuracy_2_of_3‚1 :"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.40047892928123474 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 ) Purity/4FRt0SRqqt0FBt/FRt_purity‚0.25 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9837464094161987 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.11338554322719574 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.013377204537391663 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 *"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.5 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9172576665878296 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.01274713221937418 7 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.2666666805744171 + Purity/4FRt*SRqqt0FBt/FRt_purity‚0.1875 particle/accuracy_2_of_2‚1 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.16779328882694244 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.05029762163758278 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.4074498414993286 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.03733651712536812 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.3855642080307007 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9977324604988098 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9896175265312195 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 , Purity/1FRt3SRqqt1FBt/FRt_purity‚0.28125 6 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.440972238779068 9"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.4399999976158142 ) Purity/4FRt0SRqqt*FBt/FRt_purity‚0.25 _step‚696 :"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.41308706998825073 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 + Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9375 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 + Purity/4FRt*SRqqt*FBt/FRt_purity‚0.1875 6 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.405729740858078 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.7142857313156128 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.2013375461101532 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 8"Purity/0FRt1SRqqt*FBt/event_purity‚0.311468243598938 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9992784857749939 6 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.983730137348175 :"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.40931782126426697 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 ,jet/accuracy_3_of_3‚0.024657664820551872 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.1944444626569748 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 particle/accuracy_3_of_4‚1 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8518518209457397 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 6 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.977432131767273 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.3836245536804199 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.24528302252292633 7 Purity/3FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 :"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.41276273131370544 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 *jet/accuracy_2_of_4‚0.3772428035736084 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.6666666865348816 & Purity/1FRt3SRqqt2FBt/FRt_purity‚1 trainer/global_step‚8462 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.6388888955116272 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 *"Purity/0FRt2SRqqt3FBt/SRqqt_purity‚0.5 _runtime‚9878.371978162 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.5387192964553833 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.4687499701976776 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.20000000298023224 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 ( Purity/*FRt3SRqqt2FBt/FBt_purity‚0.5 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8234399557113647 :"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.40004703402519226 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9941308498382568 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 ( Purity/1FRt*SRqqt4FBt/FRt_purity‚0.5 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 9"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.4000000059604645 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.29114383459091187 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5673704743385315 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.24000002443790436 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.5405701994895935 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 9"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.4000000059604645 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.2971656918525696 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9929742813110352 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.01848120056092739 6 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.318452388048172 7 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.2370685189962387 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 :"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.42441871762275696 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.3368113338947296 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 7 Purity/1FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9975394606590271 ;"Purity/0FRt2SRqqt1FBt/event_purity‚0.009433962404727936 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.32095444202423096 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.3416666090488434 +"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.25 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.23276010155677795 :"Purity/0FRt*SRqqt*FBt/event_purity‚0.41952061653137207 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 6validation_average_jet_accuracy‚0.4124082028865814 7 Purity/*FRt3SRqqt*FBt/FBt_purity‚0.9464285969734192 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 +"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.25 :"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.49868205189704895 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 8"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.388574481010437 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.14777515828609467 8 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.39772728085517883 :"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.49142733216285706 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 8 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.37120530009269714 ,jet/accuracy_4_of_4‚0.006763959303498268 particle/accuracy_3_of_5‚1 jet/accuracy_7_of_7‚0 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.7222222089767456 -"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.6875 5 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.46793133020401 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9891400933265686 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.24916496872901917 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 ;"Purity/0FRt3SRqqt0FBt/event_purity‚0.026143791154026985 8"Purity/*FRt0SRqqt2FBt/event_purity‚0.280161589384079 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.5293986797332764 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.2707158029079437 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 7 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.9166666865348816 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.1666666716337204 *"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.5 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.4089004397392273 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.3996860086917877 :"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.41761061549186707 :"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.41725343465805054 8 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.49709299206733704 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.05347636714577675 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.5300595164299011 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 *jet/accuracy_1_of_4‚0.9153775572776794 particle/accuracy_5_of_5‚1 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9270833730697632 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 7 Purity/0FRt3SRqqt*FBt/FBt_purity‚0.8888888955116272 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.6666666865348816 *jet/accuracy_1_of_3‚0.8645411133766174 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 jet/accuracy_4_of_7‚0 ) Purity/3FRt3SRqqt*FBt/FRt_purity‚0.25 ) Purity/1FRt1SRqqt3FBt/FRt_purity‚0.75 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9872516989707947 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.2222222238779068 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.20854607224464417 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 *"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.5 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 <"Purity/1FRt2SRqqt0FBt/event_purity‚0.0012210012646391988 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.21264366805553436 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 particle/accuracy_1_of_2‚1 7 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.2907407879829407 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 *jet/accuracy_3_of_4‚0.0647110566496849 9"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.3529411852359772 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.30963027477264404 *jet/accuracy_2_of_6‚0.5320513248443604 jet/accuracy_1_of_7‚1 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 :"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.38950979709625244 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.28395065665245056 +jet/accuracy_2_of_3‚0.24503810703754425 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5292080640792847 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.6315789222717285 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 ,"Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.375 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5415194630622864 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_2_of_5‚1 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 ( Purity/*FRt*SRqqt4FBt/FRt_purity‚0.5 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 :"Purity/2FRt*SRqqt2FBt/event_purity‚0.03875969350337982 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9984331727027893 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 particle/accuracy_1_of_5‚1 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 particle/accuracy_3_of_6‚1 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 ("Purity/4FRt*SRqqt0FBt/SRqqt_purity‚1 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.4328940212726593 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 ;"Purity/*FRt3SRqqt*FBt/event_purity‚0.008064515888690948 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 9"Purity/*FRt*SRqqt3FBt/event_purity‚0.1560283750295639 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.23357118666172028 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.24603687226772308 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 & Purity/0FRt2SRqqt2FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6448966860771179 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.5666666626930237 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.4444444477558136 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 *"Purity/*FRt2SRqqt3FBt/SRqqt_purity‚0.5 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 6 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.343071848154068 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 * Purity/*FRt*SRqqt4FBt/FBt_purity‚0.875 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.020244428887963295 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.00811338797211647 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.30672702193260193 + Purity/2FRt2SRqqt1FBt/FRt_purity‚0.1875 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9909009337425232 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.29111525416374207 :"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.43470263481140137 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.23731723427772522 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 9"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.4722222089767456 *jet/accuracy_2_of_5‚0.4661034941673279 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.2929718494415283 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.6359126567840576 +particle/sensitivity‚0.3448728621006012 ) Purity/*FRt2SRqqt2FBt/FRt_purity‚0.25 ) Purity/4FRt1SRqqt0FBt/FRt_purity‚0.25 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 ;"Purity/0FRt1SRqqt2FBt/event_purity‚0.056342363357543945 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9941520690917969 8 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.49513036012649536 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 ,jet/accuracy_4_of_5‚0.017473118379712105 particle/accuracy_1_of_1‚1 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.5177544355392456 & Purity/*FRt0SRqqt4FBt/FRt_purity‚1 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.2885582447052002 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 & Purity/2FRt2SRqqt2FBt/FRt_purity‚0 7 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.5588235259056091 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 ;"Purity/1FRt1SRqqt1FBt/event_purity‚0.041156187653541565 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.30952462553977966 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 -"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.5625 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.26181960105895996 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.5287061929702759 %jet/accuracy_1_of_5‚ 0.97802734375 particle/accuracy_2_of_6‚1 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.15930019319057465 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 6 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.287045419216156 6 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.283108115196228 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.6491227746009827 *"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.5 7 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.2611654996871948 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.3937709629535675 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 jet/accuracy_5_of_7‚0 7 Purity/3FRt1SRqqt2FBt/FRt_purity‚0.3333333432674408 epoch‚6 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.006897203624248505 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.25749990344047546 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 +"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.25 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.29240572452545166 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9900730848312378 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 :"Purity/2FRt0SRqqt2FBt/event_purity‚0.06578947603702545 particle/accuracy_3_of_3‚1 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 :"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.43209874629974365NâùÃ$”¦ <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 jet/accuracy_5_of_7‚0 7 Purity/3FRt1SRqqt2FBt/FRt_purity‚0.3333333432674408 epoch‚6 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.006897203624248505 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.25749990344047546 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 +"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.25 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.29240572452545166 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9900730848312378 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 :"Purity/2FRt0SRqqt2FBt/event_purity‚0.06578947603702545 particle/accuracy_3_of_3‚1 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 :"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.43209874629974365 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.8947368264198303 ;"Purity/1FRt2SRqqt1FBt/event_purity‚0.005747126415371895 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.24458786845207214 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9847334027290344 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.2870922386646271 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.3724512755870819 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.5097222328186035 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.29595959186553955 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 6 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.985588014Õû™ù125824 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 + Purity/3FRt1SRqqt1FBt/FRt_purity‚0.3125 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.04648250713944435 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5291425585746765 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.02327430061995983 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.3645538091659546 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.3028632402420044 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.7142857313156128 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 :"Purity/0FRt0SRqqt2FBt/event_purity‚0.36797860264778137 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.2623692452907562 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.30873072147369385 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 jet/accuracy_1_of_6‚1 particle/accuracy_5_of_6‚1 ("Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.4922527074813843 8"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.410216361284256 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9904694557189941 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9942528009414673 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 particle/accuracy_4_of_7‚1 6 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.274710476398468 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 ) Purity/4FRt1SRqqt*FBt/FRt_purity‚0.25 + Purity/1FRt0SRqqt3FBt/FRt_purity‚0.5625 8"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.696969747543335 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9890053868293762 7 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.2739463746547699 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 7 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.3194443881511688 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 & Purity/1FRt0SRqqt4FBt/FRt_purity‚1 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 particle/accuracy_5_of_7‚1 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 jet/accuracy_4_of_6‚0 6 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.949999988079071 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.4642857015132904 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 ;"Purity/*FRt1SRqqt2FBt/event_purity‚0.027118537575006485 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 jet/accuracy_6_of_7‚0 particle/accuracy_6_of_7‚1 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.5366758704185486 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.24767287075519562 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 particle/accuracy_3_of_7‚1 *"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.5 particle/accuracy_7_of_7‚1 8"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.699999988079071 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.3700917065143585 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.09598515182733536 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 +jet/accuracy_1_of_1‚0.43919119238853455 particle/accuracy_1_of_3‚1 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9703981876373291 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 +"Purity/1FRt0SRqqt3FBt/event_purity‚0.25 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.5845587253570557 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 +jet/accuracy_3_of_5‚0.08639272302389145 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 *"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.5 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9969581365585327 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5386456251144409 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.10549060255289078 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5461407899856567 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9351851940155029 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.02865961380302906 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 :"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.48337745666503906 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.21783624589443207 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 particle/accuracy_4_of_5‚1 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.19696970283985138 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.4675925672054291 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 +"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.25 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 5 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.80952388048172 +"Purity/1FRt1SRqqt3FBt/SRqqt_purity‚0.75 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.24111545085906982 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.2936336100101471 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 (particle/accuracy‚0.8564813137054443 7 Purity/*FRt3SRqqt1FBt/FBt_purity‚0.9629629850387573 particle/accuracy_4_of_6‚1 jet/accuracy_2_of_7‚0.5 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.4000000059604645 ;"Purity/*FRt3SRqqt0FBt/event_purity‚0.008064515888690948 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9680315852165222 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 ) Purity/*FRt1SRqqt3FBt/FRt_purity‚0.75 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.11004329472780228 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚1 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.3915492594242096 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9909475445747375 ;"Purity/0FRt3SRqqt*FBt/event_purity‚0.026143791154026985 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 ) Purity/3FRt3SRqqt0FBt/FRt_purity‚0.25 +"Purity/1FRt*SRqqt3FBt/SRqqt_purity‚0.75 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.19287176430225372 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 particle/accuracy_4_of_4‚1 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.28339147567749023 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.11403507739305496 9"Purity/0FRt0SRqqt3FBt/event_purity‚0.1354166567325592 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.3472222089767456 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 :"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.37006619572639465 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.18281467258930206 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.24881884455680847 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.25174012780189514 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 9"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.3840160667896271 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.5780249238014221 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 7 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.2666666805744171 particle/accuracy_6_of_6‚1 9"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.3561002314090729 7 Purity/0FRt3SRqqt1FBt/FBt_purity‚0.8888888955116272 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.1865079402923584 9"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.3912280797958374 :"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.45690351724624634 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.5556091070175171 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.25938597321510315 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.26929011940956116 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9969576597213745 particle/accuracy_1_of_6‚1 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.3571428656578064 particle/accuracy_1_of_7‚1 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9904512166976929 8 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.35273367166519165 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 8"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.439007043838501 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.24172650277614594 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 *jet/accuracy_3_of_6‚0.1538461595773697 ("Purity/4FRt*SRqqt*FBt/SRqqt_purity‚1 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 8 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.40776312351226807 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 7 Purity/1FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 6 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.932539701461792 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 <"Purity/1FRt2SRqqt*FBt/event_purity‚0.0025012025143951178 *jet/accuracy_1_of_2‚0.7292569279670715 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 ( Purity/1FRt3SRqqt2FBt/FBt_purity‚0.5 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 particle/accuracy_2_of_7‚1 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 6 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.989056408405304 *loss/val_total_loss‚38.623757255404314 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.5206896066665649 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 * Purity/0FRt0SRqqt4FBt/FBt_purity‚0.875 jet/accuracy_6_of_6‚0 :"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.35185182094573975 :"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.41118374466896057 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.3645538091659546 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.023960694670677185 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 +jet/accuracy_2_of_2‚0.10485740751028061 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.1944444626569748 & Purity/*FRt3SRqqt2FBt/FRt_purity‚1 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.14138785004615784 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 7 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.2380952537059784 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6829110383987427 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 7 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.2593677043914795 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 jet/accuracy_5_of_5‚0 particle/accuracy_2_of_4‚1 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.3402777910232544 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 jet/accuracy_5_of_6‚0 7 Purity/3FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 jet/accuracy_3_of_7‚0 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 % _timestamp‚1.7885202528992724e+09 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9739582538604736 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.25718387961387634 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.5028735995292664 +particle/specificity‚0.9511528611183167 'particle/f_score‚0.4285372197628021 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚1 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.20000000298023224 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9921626448631287 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 :"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.44581881165504456 :"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.43313494324684143 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 particle/accuracy_1_of_4‚1 9"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.5066666603088379 ("Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 :"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.42717739939689636 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.23731723427772522 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 6 Purity/1FRt3SRqqt*FBt/FBt_purity‚0.970588207244873 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 particle/accuracy_2_of_3‚1 :"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.40047892928123474 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 ) Purity/4FRt0SRqqt0FBt/FRt_purity‚0.25 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9837464094161987 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.11338554322719574 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.013377204537391663 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 *"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.5 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9172576665878296 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.01274713221937418 7 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.2666666805744171 + Purity/4FRt*SRqqt0FBt/FRt_purity‚0.1875 particle/accuracy_2_of_2‚1 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.16779328882694244 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.05029762163758278 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.4074498414993286 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.03733651712536812 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.3855642080307007 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9977324604988098 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9896175265312195 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 , Purity/1FRt3SRqqt1FBt/FRt_purity‚0.28125 6 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.440972238779068 9"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.4399999976158142 ) Purity/4FRt0SRqqt*FBt/FRt_purity‚0.25 _step‚696 :"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.41308706998825073 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 + Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9375 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 + Purity/4FRt*SRqqt*FBt/FRt_purity‚0.1875 6 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.405729740858078 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.7142857313156128 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.2013375461101532 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 8"Purity/0FRt1SRqqt*FBt/event_purity‚0.311468243598938 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9992784857749939 6 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.983730137348175 :"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.40931782126426697 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 ,jet/accuracy_3_of_3‚0.024657664820551872 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.1944444626569748 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 particle/accuracy_3_of_4‚1 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8518518209457397 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 6 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.977432131767273 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.3836245536804199 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.24528302252292633 7 Purity/3FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 :"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.41276273131370544 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 *jet/accuracy_2_of_4‚0.3772428035736084 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.6666666865348816 & Purity/1FRt3SRqqt2FBt/FRt_purity‚1 trainer/global_step‚8462 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.6388888955116272 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 *"Purity/0FRt2SRqqt3FBt/SRqqt_purity‚0.5 _runtime‚9878.371978162 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.5387192964553833 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.4687499701976776 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.20000000298023224 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 ( Purity/*FRt3SRqqt2FBt/FBt_purity‚0.5 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8234399557113647 :"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.40004703402519226 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9941308498382568 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 ( Purity/1FRt*SRqqt4FBt/FRt_purity‚0.5 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 9"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.4000000059604645 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.29114383459091187 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5673704743385315 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.24000002443790436 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.5405701994895935 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 9"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.4000000059604645 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.2971656918525696 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9929742813110352 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.01848120056092739 6 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.318452388048172 7 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.2370685189962387 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 :"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.42441871762275696 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.3368113338947296 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 7 Purity/1FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9975394606590271 ;"Purity/0FRt2SRqqt1FBt/event_purity‚0.009433962404727936 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.32095444202423096 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.3416666090488434 +"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.25 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.23276010155677795 :"Purity/0FRt*SRqqt*FBt/event_purity‚0.41952061653137207 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 6validation_average_jet_accuracy‚0.4124082028865814 7 Purity/*FRt3SRqqt*FBt/FBt_purity‚0.9464285969734192 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 +"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.25 :"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.49868205189704895 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 8"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.388574481010437 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.14777515828609467 8 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.39772728085517883 :"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.49142733216285706 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 8 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.37120530009269714 ,jet/accuracy_4_of_4‚0.006763959303498268 particle/accuracy_3_of_5‚1 jet/accuracy_7_of_7‚0 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.7222222089767456 -"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.6875 5 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.46793133020401 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9891400933265686 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.24916496872901917 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 ;"Purity/0FRt3SRqqt0FBt/event_purity‚0.026143791154026985 8"Purity/*FRt0SRqqt2FBt/event_purity‚0.280161589384079 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.5293986797332764 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.2707158029079437 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 7 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.9166666865348816 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.1666666716337204 *"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.5 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.4089004397392273 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.3996860086917877 :"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.41761061549186707 :"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.41725343465805054 8 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.49709299206733704 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.05347636714577675 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.5300595164299011 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 *jet/accuracy_1_of_4‚0.9153775572776794 particle/accuracy_5_of_5‚1 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9270833730697632 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 7 Purity/0FRt3SRqqt*FBt/FBt_purity‚0.8888888955116272 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.6666666865348816 *jet/accuracy_1_of_3‚0.8645411133766174 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 jet/accuracy_4_of_7‚0 ) Purity/3FRt3SRqqt*FBt/FRt_purity‚0.25 ) Purity/1FRt1SRqqt3FBt/FRt_purity‚0.75 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9872516989707947 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.2222222238779068 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.20854607224464417 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 *"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.5 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 <"Purity/1FRt2SRqqt0FBt/event_purity‚0.0012210012646391988 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.21264366805553436 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 particle/accuracy_1_of_2‚1 7 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.2907407879829407 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 *jet/accuracy_3_of_4‚0.0647110566496849 9"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.3529411852359772 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.30963027477264404 *jet/accuracy_2_of_6‚0.5320513248443604 jet/accuracy_1_of_7‚1 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 :"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.38950979709625244 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.28395065665245056 +jet/accuracy_2_of_3‚0.24503810703754425 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5292080640792847 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.6315789222717285 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 ,"Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.375 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5415194630622864 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_2_of_5‚1 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 ( Purity/*FRt*SRqqt4FBt/FRt_purity‚0.5 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 :"Purity/2FRt*SRqqt2FBt/event_purity‚0.03875969350337982 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9984331727027893 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 particle/accuracy_1_of_5‚1 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 particle/accuracy_3_of_6‚1 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 ("Purity/4FRt*SRqqt0FBt/SRqqt_purity‚1 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.4328940212726593 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 ;"Purity/*FRt3SRqqt*FBt/event_purity‚0.008064515888690948 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 9"Purity/*FRt*SRqqt3FBt/event_purity‚0.1560283750295639 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.23357118666172028 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.24603687226772308 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 & Purity/0FRt2SRqqt2FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6448966860771179 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.5666666626930237 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.4444444477558136 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 *"Purity/*FRt2SRqqt3FBt/SRqqt_purity‚0.5 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 6 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.343071848154068 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 * Purity/*FRt*SRqqt4FBt/FBt_purity‚0.875 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.020244428887963295 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.00811338797211647 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.30672702193260193 + Purity/2FRt2SRqqt1FBt/FRt_purity‚0.1875 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9909009337425232 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 ("Purity/3FRt0SRqqt*FBt/eÛ L“ vent_purity‚0 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.29111525416374207 :"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.43470263481140137 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.23731723427772522 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 9"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.4722222089767456 *jet/accuracy_2_of_5‚0.4661034941673279 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.2929718494415283 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.6359126567840576 +particle/sensitivity‚0.3448728621006012 ) Purity/*FRt2SRqqt2FBt/FRt_purity‚0.25 ) Purity/4FRt1SRqqt0FBt/FRt_purity‚0.25 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 ;"Purity/0FRt1SRqqt2FBt/event_purity‚0.056342363357543945 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9941520690917969 8 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.49513036012649536 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 ,jet/accuracy_4_of_5‚0.017473118379712105 particle/accuracy_1_of_1‚1 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.5177544355392456 & Purity/*FRt0SRqqt4FBt/FRt_purity‚1 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.2885582447052002 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 & Purity/2FRt2SRqqt2FBt/FRt_purity‚0 7 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.5588235259056091 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 ;"Purity/1FRt1SRqqt1FBt/event_purity‚0.041156187653541565 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.30952462553977966 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 -"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.5625 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.26181960105895996 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.5287061929702759 %jet/accuracy_1_of_5‚ 0.97802734375 particle/accuracy_2_of_6‚1 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.15930019319057465 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 6 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.287045419216156 6 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.283108115196228 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.6491227746009827 *"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.5 7 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.2611654996871948 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.3937709629535675 ("Purity/3FRt1SRqqt0FBt/event_purity‚0¸bf2’Ä$‚(:‡ ÃÎêÔ£’œÿ gpu.0.gpu‚49.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚159.696' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚39.924 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ùá³íkÅ$‚(:à ÃÎêÔªªëÿ network.sent‚ 2.1845134e+07 disk.sda1.out‚0 disk.md0p1.out‚ 1092.90625 disk.sdb.in‚3.1875! disk.sdb.out‚1133.91552734375! proc.memory.rssMB‚ 3001.234375 disk.sda2.in‚ 6.08203125 disk.rbd0.in‚0# disk.nvme1n1.out‚51263.76953125 disk.md0.in‚ 9.26953125 disk.rbd2.in‚0 disk.dm-0.out‚ 108278.625 network.recv‚ 3.165586e+06 disk.sdb1.out‚0 disk.md0.out‚ 1092.90625 disk.rbd2.out‚ 2.74609375 disk.sdb2.in‚3.1875 disk.dm-0.in‚ 1116.9140625 disk.sda.in‚ 6.08203125( disk./.usagePercent‚75.8380752117169" disk.sdb2.out‚1133.91552734375 disk.nvme0n1.in‚ 618.11328125% disk./.usageGB‚2712.3369102478027" disk.sda2.out‚1133.91552734375 disk.sda1.in‚0) proc.memory.percent‚18.31808090209961 proc.cpu.threads‚17 disk.rbd0.out‚ 2.74609375 disk.sdb1.in‚0# disk.nvme0n1.out‚57014.85546875$ memory_percent‚92.11504459381104 cpu‚25.22013130620558 disk.nvme1n1.in‚ 498.80078125 disk.md0p1.in‚ 9.26953125! disk.sda.out‚1133.91552734375) proc.memory.availableMB‚ 1291.87109375. ‘Æ$‚(:† ÒÎêÔŽä ÿ gpu.0.gpu‚50.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚172.22' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.055 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÛžmÇ$‚(:â ÒÎêÔ»Ÿöÿ disk.sda2.in‚ 6.08203125 disk.sdb2.in‚3.1875! disk.sdb2.out‚1135.4462890625 disk.sda1.in‚0 disk.nvme0n1.out‚ 57076.40625$ memory_percent‚92.11413860321045) proc.memory.availableMB‚ 1292.01953125! proc.memory.rssMB‚ 3001.234375 disk.sdb.out‚1135.4462890625 disk.sdb1.out‚0! disk.dm-0.out‚108388.96484375 proc.cpu.threads‚17 disk.md0p1.out‚ 1094.3671875 network.recv‚ 3.171722e+06) disk./.usagePercent‚75.83642986634838 disk.sdb.in‚3.1875 disk.sda.in‚ 6.08203125 disk.sda.out‚1135.4462890625! disk.sda2.out‚1135.4462890625 disk.rbd0.out‚ 2.74609375# disk.nvme1n1.out‚51312.55859375 disk.rbd2.out‚ 2.74609375 disk.md0p1.in‚ 9.26953125) proc.memory.percent‚18.31808090209961 disk.sdb1.in‚0 disk.sda1.out‚0 disk.dm-0.in‚ 1116.9140625 disk.nvme0n1.in‚ 618.11328125 network.sent‚ 2.1970981e+07$ disk./.usageGB‚2712.278064727783 disk.md0.in‚ 9.26953125 disk.rbd2.in‚0 disk.rbd0.in‚0 disk.nvme1n1.in‚ 498.80078125 disk.md0.out‚ 1094.3671875 cpu‚25.21984851402938rNƒN“È$‚(:ˆ áÎêÔþ gpu.0.gpu‚55.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚37.0 gpu.0.powerWatts‚116.306' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚29.0765 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0YfhæjÉ$‚(:ß áÎêÔ–ëòÿ disk.md0p1.out‚ 1096.2890625 disk.rbd2.in‚0 disk.sda1.in‚0! disk.dm-0.out‚108463.01953125$ memory_percent‚92.11716651916504( proc.memory.availableMB‚ 1291.5234375! proc.memory.rssMB‚ 3001.234375 network.sent‚ 2.1976236e+07 network.recv‚ 3.175198e+06 disk.sdb2.in‚3.1875 disk.md0.in‚ 9.26953125% disk./.usageGB‚2712.4306716918945 disk.sdb.out‚1137.435546875 disk.sda2.in‚ 6.08203125 disk.sda2.out‚1137.435546875 disk.nvme1n1.in‚ 498.80078125 disk.md0.out‚ 1096.2890625 cpu‚25.219542136256564 proc.cpu.threads‚17 disk.rbd0.out‚ 2.74609375 disk.sdb1.out‚0( disk./.usagePercent‚75.8406968209359 disk.rbd2.out‚ 2.74609375 disk.sdb2.out‚1137.435546875# disk.nvme0n1.out‚57076.41015625) proc.memory.percent‚18.31808090209961 disk.rbd0.in‚0 disk.sdb1.in‚0 disk.dm-0.in‚ 1116.9140625 disk.nvme0n1.in‚ 618.11328125 disk.sdb.in‚3.1875 disk.sda.in‚ 6.08203125! disk.nvme1n1.out‚ 51386.609375 disk.sda1.out‚0 disk.md0p1.in‚ 9.26953125 disk.sda.out‚1137.435546875–dLD2Ê$- _wandbruntime‚9919  _runtime‚9919éʦ¦ÈË$ trainer/global_step‚8499 $ _timestamp‚1.788520293944212e+09 _runtime‚9919.414818141 _step‚697 ' lr-AdamW/pg1‚0.00021057651245056324 ' lr-AdamW/pg2‚0.00021057651245056324¬ï8ùÍÌ$Ç _runtime‚9919.414818141 _step‚697 ' lr-AdamW/pg1‚0.00021057651245056324 ' lr-AdamW/pg2‚0.00021057651245056324 trainer/global_step‚8499 $ _timestamp‚1.788520293944212e+09¹m¥&^2Í$- _wandbruntime‚9919  _runtime‚9919ÐT¶Y(Î$úO OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 213465936 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚710185261596160 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚710185261596160 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚793790856448000 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 % _timestamp‚1.7885202939451413e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177047040 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚710184344185344 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 _step‚698 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407872 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10103963773952 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917410816 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9676495535104 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 _runtime‚9919.415638175 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 175880370 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9677394942976 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1212 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9641325130030 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 160534113 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 244411392 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚783686916614144 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚783686892674048 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚684800986825264 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007241 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚51169018 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚700507848650240 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 109365093 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450780 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214665 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955591168 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚710184344185344 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1187 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚700507866653184 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 109365095 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9640427718610 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26088 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955591168 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 175880372 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚621 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚694441397504514 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚700507848650240 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9677394942976 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚8499 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 175880372 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407872 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 213492024 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410816 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚684800969785904 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚694442311955294 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411420 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚700507866653184 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 399863808Ò^(Ï$ÿO OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 213465936 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚710185261596160 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚710185261596160 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚793790856448000 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 % _timestamp‚1.7885202939451413e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177047040 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚710184344185344 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 _step‚698 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407872 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10103963773952 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917410816 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9676495535104 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 _runtime‚9919.415638175 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 175880370 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9677394942976 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1212 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9641325130030 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 160534113 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 244411392 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚783686916614144 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚783686892674048 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚684800986825264 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007241 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚51169018 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚700507848650240 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 109365093 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450780 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214665 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955591168 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚710184344185344 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1187 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚700507866653184 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 109365095 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9640427718610 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26088 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955591168 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 175880372 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚621 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚694441397504514 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚700507848650240 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9677394942976 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚8499 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 175880372 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407872 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 213492024 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410816 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚684800969785904 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚694442311955294 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411420 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚700507866653184 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 399863808ºâ †i2Ð$- _wandbruntime‚9920  _runtime‚9920ýáùÏÜÑ$„N ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31686 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚783776235834880 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚710266004708352 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚700587496948736 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚694521400710948 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9641536379928 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 213490527 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 160552135 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411628 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9677608338432 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚51174058 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917424128 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 175900270 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 175900270 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 160551527 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚793881351064064 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚684878966919392 trainer/global_step‚8499 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899421184 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177047040 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 389416911 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 399863808 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 % _timestamp‚1.7885202949883437e+09 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 389416911 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚700587478945792 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899421184 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚694520486259960 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9642433791556 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007241 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚710265087284224 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9677608338432 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9678507759616 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚710266004708352 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 109377469 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 213490527 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10105353335808 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 213516641 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 389390795 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26114 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚700587496948736 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _runtime‚9920.459689016 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10105139169280 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450988 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 175900268 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177047040 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 389416911 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚700587478945792 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899421184 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚694520486259960 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9642433791556 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007241 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚710265087284224 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9677608338432 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9678507759616 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚710266004708352 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 109377469 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 213490527 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10105353335808 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 213516641 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 389390795 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26114 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚700587496948736 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _runtime‚9920.459689016 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10105139169280 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450988 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 175900268 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177047040 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚51174058 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917424128 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 175900270 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 175900270 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 160551527 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚793881351064064 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚684878966919392 trainer/global_step‚8499 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899421184 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177047040 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 389416911 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 399863808 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 % _timestamp‚1.7885202949883437e+09 FDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1212 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚51451372 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚798314591996928 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9696771787138 TDDeviceStatsMonitor.on_trai½÷u˜ n_batch_start/active_bytes.large_pool.peak‚ 12835443712 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407872 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚51450779 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 214724926 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚714221513543168 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9732150115328 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚704489381430784 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚698395848868022 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214665 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 176875270 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 161434672 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚704489381430784 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 176875270 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚788152853710848 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955591168 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 _step‚702 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚714222430953984 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1187 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 401960960 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 11827937281?º(ã$ÿO aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚788152853710848 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955591168 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 _step‚702 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚714222430953984 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1187 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 401960960 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 391574108 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1212 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚51451372 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚798314591996928 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9696771787138 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407872 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚51450779 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 214724926 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚714221513543168 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9732150115328 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚704489381430784 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚698395848868022 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214665 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 176875270 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 161434672 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚704489381430784 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 176875270 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096¾û°Ä2ä$- _wandbruntime‚9977  _runtime‚9977ch¨G 'å$„N N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177047040 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚714302256642048 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10163122802688 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚698474937623468 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 176895170 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 109996271 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚798405058499072 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 214749543 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177047040 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 391618597 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1187 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450988 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚688777954586432 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411628 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 161452637 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 _step‚703 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31686 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚704569011726336 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9734162339840 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 161453254 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 401960960 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10162885567488 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚714303174066176 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 214749543 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚798405319674368 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚788242196871680 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26114 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899421184 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 391644713 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9697880448664 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚714303174066176 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9733262918656 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 261175296 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚8549 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 176895170 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 % _timestamp‚1.7885203516889274e+09 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚704568993723392 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9734162339840 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 109996269 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955591168 _runtime‚9977.159993987 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899421184 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9696983037036 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚788242172931584 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007241 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9733262918656 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 391644713 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 161453254 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 401960960 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10162885567488 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚714303174066176 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 214749543 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚798405319674368 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚788242196871680 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26114 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899421184 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 391644713 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9697880448664 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚714303174066176 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9733262918656 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 261175296 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚8549 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 176895170 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 % _timestamp‚1.7885203516889274e+09 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚704568993723392 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9734162339840 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 109996269 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955591168 _runtime‚9977.159993987 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899421184 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9696983037036 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚788242172931584 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007241 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9733262918656 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 391644713 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177047040 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚714302256642048 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10163122802688 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚698474937623468 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 176895170 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 109996271 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚798405058499072 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 214749543 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177047040 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 391618597 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1187 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450988 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚688777954586432 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411628 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 161452637 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 _step‚703 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31686 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚704569011726336 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9734162339840¿ê)Ž2ç$- _wandbruntime‚9977  _runtime‚9977™ñXè$Ò /loss/FRt4/assignment_loss‚2.831918625270619 1loss/SRqqt4/detection_loss‚2.3177797613721904 0loss/FRt1/assignment_loss‚2.3099948040076663 .loss/FRt1/detection_loss‚2.194645328621104 1loss/SRqqt1/assignment_loss‚1.083920524460781 /loss/FRt3/detection_loss‚2.3140012872490017 -loss/FBt2/detection_loss‚0.65257377970184 -loss/FBt3/detection_loss‚3.27808596028222 % _timestamp‚1.7885203516907806e+09 1loss/FBt2/assignment_loss‚0.09669625605206617 0loss/FBt3/assignment_loss‚0.3517967875056153 1loss/FBt4/assignment_loss‚0.24119809702110614 /loss/FRt4/detection_loss‚3.4767789244651794 .loss/FBt1/detection_loss‚2.224864525454385 _runtime‚9977.160967321 2loss/SRqqt2/assignment_loss‚0.8262406416958378 2loss/SRqqt4/assignment_loss‚1.7209953905055018 1loss/FBt1/assignment_loss‚0.15350952208973467 /loss/FRt2/detection_loss‚1.5486822166903453 1loss/SRqqt1/detection_loss‚1.8006748686234155 1loss/SRqqt2/detection_loss‚1.2474950755675953 /loss/FBt4/detection_loss‚2.1369739294052126 epoch‚7 0loss/SRqqt3/detection_loss‚1.625068449570487 %loss/total_loss‚38.88807698782827 trainer/global_step‚8549 0loss/FRt2/assignment_loss‚0.9890454628250815 /loss/FRt3/assignment_loss‚2.370749129490419 2loss/SRqqt3/assignment_loss‚1.0943876399008634 _step‚704o­ ]é$× /loss/FRt3/assignment_loss‚2.370749129490419 2loss/SRqqt3/assignment_loss‚1.0943876399008634 _step‚704 /loss/FRt4/assignment_loss‚2.831918625270619 1loss/SRqqt4/detection_loss‚2.3177797613721904 0loss/FRt1/assignment_loss‚2.3099948040076663 .loss/FRt1/detection_loss‚2.194645328621104 1loss/SRqqt1/assignment_loss‚1.083920524460781 /loss/FRt3/detection_loss‚2.3140012872490017 -loss/FBt2/detection_loss‚0.65257377970184 -loss/FBt3/detection_loss‚3.27808596028222 % _timestamp‚1.7885203516907806e+09 1loss/FBt2/assignment_loss‚0.09669625605206617 0loss/FBt3/assignment_loss‚0.3517967875056153 1loss/FBt4/assignment_loss‚0.24119809702110614 /loss/FRt4/detection_loss‚3.4767789244651794 .loss/FBt1/detection_loss‚2.224864525454385 _runtime‚9977.160967321 2loss/SRqqt2/assignment_loss‚0.8262406416958378 2loss/SRqqt4/assignment_loss‚1.7209953905055018 1loss/FBt1/assignment_loss‚0.15350952208973467 /loss/FRt2/detection_loss‚1.5486822166903453 1loss/SRqqt1/detection_loss‚1.8006748686234155 1loss/SRqqt2/detection_loss‚1.2474950755675953 /loss/FBt4/detection_loss‚2.1369739294052126 epoch‚7 0loss/SRqqt3/detection_loss‚1.625068449570487 %loss/total_loss‚38.88807698782827 trainer/global_step‚8549 0loss/FRt2/assignment_loss‚0.9890454628250815ÀØœw'“ê$‚(:ˆ ¬ÏêÔý’ÿ gpu.0.gpu‚41.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚170.718' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.6795 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÝTˆ(të$‚(:é ¬ÏêÔ‘ÔÑÿ disk.md0p1.out‚ 1104.16796875 network.sent‚ 2.2105192e+07 disk.sdb1.in‚0" disk.sdb2.out‚1145.59814453125! disk.nvme0n1.out‚ 57076.984375! disk.sda.out‚1145.59814453125 disk.sda2.in‚ 6.08203125% disk./.usageGB‚2713.8710403442383 disk.sdb1.out‚0 disk.sdb2.in‚3.1875 disk.nvme0n1.in‚ 618.11328125 disk.sdb.in‚3.1875 disk.sda.in‚ 6.08203125 proc.cpu.threads‚17 disk.md0.out‚ 1104.16796875 disk.sda1.out‚0 disk.dm-0.in‚ 1116.9140625! disk.dm-0.out‚109361.36328125! disk.sdb.out‚1145.59814453125 disk.rbd0.in‚0 disk.rbd0.out‚ 2.74609375# disk.nvme1n1.out‚52284.37890625) disk./.usagePercent‚75.88097013129655 disk.rbd2.in‚0 disk.nvme1n1.in‚ 498.80078125' proc.memory.availableMB‚ 1291.328125 cpu‚25.218173946637705 network.recv‚ 3.197502e+06 disk.rbd2.out‚ 2.74609375 disk.sda1.in‚0" disk.sda2.out‚1145.59814453125$ memory_percent‚92.11835861206055! proc.memory.rssMB‚ 3001.234375) proc.memory.percent‚18.31808090209961 disk.md0.in‚ 9.26953125 disk.md0p1.in‚ 9.26953125uцû“ì$‚(:ˆ »ÏêÔ®®‡þ gpu.0.gpu‚46.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚37.0 gpu.0.powerWatts‚122.486' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚30.6215 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 }ßlí$‚(:á »ÏêÔáœÖÿ network.recv‚ 3.202819e+06 disk.sdb1.out‚0 disk.sdb2.in‚3.1875 disk.sda1.out‚0 disk.sda2.in‚ 6.08203125( proc.memory.availableMB‚ 1291.3515625 cpu‚25.217897425960278# disk.nvme1n1.out‚52284.51171875! disk.sdb.out‚1147.15771484375 network.sent‚ 2.2212698e+07 disk.rbd2.in‚0" disk.sdb2.out‚1147.15771484375 disk.sdb.in‚3.1875 disk.rbd0.out‚ 2.74609375 disk.sda1.in‚0 disk.md0p1.in‚ 9.26953125 disk.md0p1.out‚ 1105.671875! proc.memory.rssMB‚ 3001.234375% disk./.usageGB‚2713.8710975646973 disk.rbd2.out‚ 2.74609375" disk.sda2.out‚1147.15771484375) proc.memory.percent‚18.31808090209961 disk.rbd0.in‚0 disk.nvme1n1.in‚ 498.80078125 disk.sdb1.in‚0 disk.dm-0.out‚109361.5 disk.nvme0n1.in‚ 618.11328125# disk.nvme0n1.out‚57076.98828125$ memory_percent‚92.11821556091309) disk./.usagePercent‚75.88097173120454 disk.md0.in‚ 9.26953125 disk.md0.out‚ 1105.671875 disk.dm-0.in‚ 1116.9140625 disk.sda.in‚ 6.08203125! disk.sda.out‚1147.15771484375 proc.cpu.threads‚17v’’î$‚(:‡ ÊÏêÔíþàý gpu.0.gpu‚56.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚168.992' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.248 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0úígmï$‚(:â ÊÏêÔìñÑþ disk.nvme0n1.in‚ 618.11328125 network.sent‚ 2.2218167e+07 disk.dm-0.in‚ 1116.9140625! disk.dm-0.out‚109892.25390625 disk.md0p1.in‚ 9.26953125! proc.memory.rssMB‚ 3001.234375 disk.sda.out‚1149.0693359375 disk.rbd0.out‚ 2.74609375 disk.nvme1n1.in‚ 498.80078125 disk.rbd2.in‚0 proc.cpu.threads‚17# disk.nvme0n1.out‚57077.27734375 disk.sda1.in‚0! disk.sda2.out‚1149.0693359375 disk.rbd0.in‚0 disk.sdb1.out‚0$ disk./.usageGB‚2713.871120452881 disk.sda.in‚ 6.08203125 cpu‚25.217627186232807 disk.sdb.out‚1149.0693359375 network.recv‚ 3.20688e+06 disk.sda1.out‚0) disk./.usagePercent‚75.88097237116773 disk.sdb.in‚3.1875 disk.rbd2.out‚ 2.74609375$ memory_percent‚92.11695194244385 disk.sda2.in‚ 6.08203125 disk.md0.in‚ 9.26953125 disk.sdb2.in‚3.1875! disk.sdb2.out‚1149.0693359375) proc.memory.availableMB‚ 1291.55859375) proc.memory.percent‚18.31808090209961" disk.nvme1n1.out‚ 52814.9765625 disk.md0.out‚ 1107.515625 disk.md0p1.out‚ 1107.515625 disk.sdb1.in‚0A©œ4ð$/ _wandbruntime‚10031  _runtime‚10031UšrÌÉñ$à ' lr-AdamW/pg1‚0.00020001086382439933 ' lr-AdamW/pg2‚0.00020001086382439933 trainer/global_step‚8599 $ _timestamp‚1.788520405636547e+09 _runtime‚10031.106879487 _step‚705ð:¼šÎò$È ' lr-AdamW/pg1‚0.00020001086382439933 ' lr-AdamW/pg2‚0.00020001086382439933 trainer/global_step‚8599 $ _timestamp‚1.788520405636547e+09 _runtime‚10031.106879487 _step‚705Áµ›4ó$/ _wandbruntime‚10031  _runtime‚10031‚¾{ä(ô$ûO H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955591168 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚708470896208384 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚792618814747648 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10220047141376 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411420 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 110602695 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚718258682900992 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9788704103424 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10219832961536 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9751321032826 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚692598996238064 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9752218444246 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917410816 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 215931740 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9787804695552 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955591168 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 177870170 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚802838647709184 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 214179840 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚802838885829120 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚708470878205440 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 177870172 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 238119936 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1212 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚708470878205440 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9787804695552 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚718258682900992 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407872 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26088 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚692598979198704 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚51733858 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214665 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177047040 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 393801910 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚708470896208384 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 215931740 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 162335971 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 110602693 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚580 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 401960960 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚718258682900992 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407872 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26088 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚692598979198704 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚51733858 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214665 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177047040 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 393801910 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚708470896208384 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 215931740 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 162335971 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 110602693 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚580 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 401960960 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955591168 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚708470896208384 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚792618814747648 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10220047141376 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411420 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 110602695 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚718258682900992 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9788704103424 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10219832961536 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9751321032826 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚692598996238064 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9752218444246 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917410816 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 215931740 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9787804695552 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955591168 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 177870170 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚802838647709184 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 214179840 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚802838885829120 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚708470878205440 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 177870172 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 238119936 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564±þ«¯4ö$/ _wandbruntime‚10032  _runtime‚10032½Q)Q÷$…N YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9789816920064 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚51739551 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 222555136 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚708550526503936 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917424128 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9753327105772 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 393846399 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31686 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚607 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450988 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899421184 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 215982445 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899421184 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 177890070 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9789816920064 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 110615071 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007241 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 % _timestamp‚1.7885204065941446e+09 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10221222083072 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚702429388986976 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 177890068 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚708550508500992 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9788917498880 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 TFDevióZq׺ceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 246495232 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 110615069 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚692676976332192 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26114 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 393872515 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 215956331 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9752429694144 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚708550526503936 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 177890068 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 trainer/global_step‚8599 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 393872515 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 215956331 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚51738946 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955591168 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 162354622 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚718340343424000 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚802929379991552 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9788917498880 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26114 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚702430303437964 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 401960960 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚792708133968384 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚718340343424000 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚605 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1187 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 177890070 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10220999527936 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚692676959292832 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411628 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 393846399 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 162354015 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 _runtime‚10032.064991168 _step‚707ãüCæ'ø$ŠN S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚702429388986976 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 177890068 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚708550508500992 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9788917498880 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 246495232 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 110615069 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚692676976332192 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26114 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 393872515 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 215956331 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9752429694144 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚708550526503936 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 177890068 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 trainer/global_step‚8599 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 393872515 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 215956331 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚51738946 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955591168 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 162354622 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚718340343424000 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚802929379991552 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9788917498880 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26114 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚702430303437964 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 401960960 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚792708133968384 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚718340343424000 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚605 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1187 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 177890070 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10220999527936 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚692676959292832 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411628 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 393846399 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 162354015 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 _runtime‚10032.064991168 _step‚707 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9789816920064 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚51739551 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 222555136 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚708550526503936 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917424128 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9753327105772 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 393846399 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31686 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚607 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450988 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899421184 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 215982445 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899421184 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 177890070 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9789816920064 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 110615071 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007241 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 % _timestamp‚1.7885204065941446e+09 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10221222083072 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360ÃV­ÂS4ù$/ _wandbruntime‚10032  _runtime‚10032Žè^ú$Ø /loss/FBt2/detection_loss‚0.5830461174720921 0loss/FBt4/assignment_loss‚0.1734154469657789 0loss/FRt2/assignment_loss‚1.0602792392174403 1loss/FBt1/assignment_loss‚0.20801183872663387 1loss/SRqqt4/detection_loss‚2.7993974506855013 .loss/FRt1/detection_loss‚2.094881536156107 /loss/FBt3/detection_loss‚2.6898843462650595 /loss/FRt2/detection_loss‚1.2360940668111047 2loss/SRqqt3/assignment_loss‚1.2427113176877234 0loss/SRqqt3/detection_loss‚2.058507954349389 /loss/FBt1/detection_loss‚2.4334384128451347 epoch‚7 0loss/FRt3/assignment_loss‚2.6967006087303163 /loss/FRt3/detection_loss‚2.5598760566674175 /loss/FRt4/detection_loss‚3.2274513708220587 &loss/total_loss‚39.145201882703326 _step‚708 /loss/FRt4/assignment_loss‚3.018944170739916 2loss/SRqqt1/assignment_loss‚1.3293517982681928 2loss/SRqqt2/assignment_loss‚0.9409204622403391 0loss/FBt2/assignment_loss‚0.2030835813158192 2loss/SRqqt4/assignment_loss‚1.5830765277147294 % _timestamp‚1.7885204065963938e+09 0loss/FRt1/assignment_loss‚2.2528672907087537 trainer/global_step‚8599 0loss/FBt3/assignment_loss‚0.1544726007112051 1loss/SRqqt1/detection_loss‚1.4252919082419346 1loss/SRqqt2/detection_loss‚0.6497687347031929 .loss/FBt4/detection_loss‚2.523729044657487 _runtime‚10032.066518274†U^cû$Ý 0loss/FBt2/assignment_loss‚0.2030835813158192 2loss/SRqqt4/assignment_loss‚1.5830765277147294 % _timestamp‚1.7885204065963938e+09 0loss/FRt1/assignment_loss‚2.2528672907087537 trainer/global_step‚8599 0loss/FBt3/assignment_loss‚0.1544726007112051 1loss/SRqqt1/detection_loss‚1.4252919082419346 1loss/SRqqt2/detection_loss‚0.6497687347031929 .loss/FBt4/detection_loss‚2.523729044657487 _runtime‚10032.066518274 /loss/FBt2/detection_loss‚0.5830461174720921 0loss/FBt4/assignment_loss‚0.1734154469657789 0loss/FRt2/assignment_loss‚1.0602792392174403 1loss/FBt1/assignment_loss‚0.20801183872663387 1loss/SRqqt4/detection_loss‚2.7993974506855013 .loss/FRt1/detection_loss‚2.094881536156107 /loss/FBt3/detection_loss‚2.6898843462650595 /loss/FRt2/detection_loss‚1.2360940668111047 2loss/SRqqt3/assignment_loss‚1.2427113176877234 0loss/SRqqt3/detection_loss‚2.058507954349389 /loss/FBt1/detection_loss‚2.4334384128451347 epoch‚7 0loss/FRt3/assignment_loss‚2.6967006087303163 /loss/FRt3/detection_loss‚2.5598760566674175 /loss/FRt4/detection_loss‚3.2274513708220587 &loss/total_loss‚39.145201882703326 _step‚708 /loss/FRt4/assignment_loss‚3.018944170739916 2loss/SRqqt1/assignment_loss‚1.3293517982681928 2loss/SRqqt2/assignment_loss‚0.9409204622403391ÄV[”ü$‚(:‰ ÙÏêÔ—²« gpu.0.gpu‚12.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚159.079' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚39.76975 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ó?¢hý$‚(:Ý ÙÏêÔý˼‚ disk.rbd0.out‚ 2.74609375 disk.md0.in‚ 9.26953125 disk.dm-0.in‚ 1116.9140625 network.sent‚ 2.2223466e+07) disk./.usagePercent‚75.88097237116773! disk.sdb2.out‚1150.8310546875 cpu‚25.217357190137424$ disk./.usageGB‚2713.871120452881# disk.nvme1n1.out‚52815.11328125 disk.rbd2.out‚ 2.74609375$ memory_percent‚92.11578369140625 network.recv‚ 3.210393e+06 disk.sdb.in‚3.1875 disk.sda2.in‚ 6.08203125! disk.sda2.out‚1150.8310546875 disk.nvme1n1.in‚ 498.80078125 disk.sdb2.in‚3.1875! proc.memory.rssMB‚ 3001.234375 disk.md0.out‚ 1109.2265625 disk.rbd2.in‚0 disk.nvme0n1.in‚ 618.11328125 disk.nvme0n1.out‚ 57077.28125 disk.sdb.out‚1150.8310546875 disk.sda.in‚ 6.08203125 disk.rbd0.in‚0 disk.sdb1.in‚0 disk.md0p1.out‚ 1109.2265625) proc.memory.percent‚18.31808090209961 disk.sda.out‚1150.8310546875 disk.sdb1.out‚0! disk.dm-0.out‚109892.39453125 disk.sda1.in‚0 disk.sda1.out‚0 disk.md0p1.in‚ 9.26953125# proc.memory.availableMB‚1291.75 proc.cpu.threads‚17ãЩ’þ$‚(:‡ èÏêÔñþ… gpu.0.gpu‚56.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚37.0 gpu.0.powerWatts‚120.65' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚30.1625 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0vëªÇmÿ$‚(:â èÏêÔ³û׈ disk.rbd0.in‚0! proc.memory.rssMB‚ 3001.234375 proc.cpu.threads‚17 network.sent‚ 2.2331132e+07 disk.md0.in‚ 9.26953125) proc.memory.percent‚18.31808090209961! disk.sdb.out‚1152.27099609375 disk.nvme1n1.in‚ 498.80078125 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.dm-0.in‚ 1116.9140625# proc.memory.availableMB‚1291.75 disk.md0.out‚ 1110.60546875 disk.sda1.out‚0 disk.md0p1.out‚ 1110.60546875 disk.rbd0.out‚ 2.74609375 disk.rbd2.in‚0" disk.sdb2.out‚1152.27099609375 cpu‚25.21700174466378! disk.sda.out‚1152.27099609375# disk.nvme1n1.out‚53190.62109375! disk.dm-0.out‚110268.27734375 disk.md0p1.in‚ 9.26953125 network.recv‚ 3.21607e+06 disk.sdb.in‚3.1875 disk.sda.in‚ 6.08203125 disk.sda2.in‚ 6.08203125 disk.sda1.in‚0" disk.sda2.out‚1152.27099609375 disk.rbd2.out‚ 2.74609375 disk.sdb2.in‚3.1875$ memory_percent‚92.11578369140625) disk./.usagePercent‚75.86614559725801% disk./.usageGB‚2713.3408432006836 disk.nvme0n1.in‚ 618.11328125 disk.nvme0n1.out‚ 57077.656250TÍ”€%‚(:‰ ÷ÏêÔ–Ñ¢þ gpu.0.gpu‚55.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚39.0 gpu.0.powerWatts‚175.235' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.80875 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¢¢úéh%‚(:Ý ÷ÏêÔç‹¥ disk.sdb.out‚1153.822265625 disk.sda.out‚1153.822265625" disk.nvme1n1.out‚ 53190.7890625 disk.rbd2.in‚0 proc.cpu.threads‚17 disk.sda2.in‚ 6.08203125 disk.sda2.out‚1153.822265625# disk.nvme0n1.out‚57077.66015625) disk./.usagePercent‚75.83666163968468 disk.sda1.in‚0 disk.sda1.out‚0! disk.dm-0.out‚110268.44921875 disk.rbd0.out‚ 2.74609375 disk.md0.out‚ 1112.09375 disk.sdb2.out‚1153.822265625 disk.md0p1.out‚ 1112.09375 cpu‚25.216692341208326% disk./.usageGB‚2712.2863540649414 disk.rbd0.in‚0 disk.nvme1n1.in‚ 498.80078125 disk.md0p1.in‚ 9.26953125$ memory_percent‚92.11726188659668( proc.memory.availableMB‚ 1291.5078125 network.sent‚ 2.2339581e+07 network.recv‚ 3.220971e+06 disk.sdb.in‚3.1875 disk.sda.in‚ 6.08203125! proc.memory.rssMB‚ 3001.234375) proc.memory.percent‚18.31808090209961 disk.md0.in‚ 9.26953125 disk.rbd2.out‚ 2.74609375 disk.sdb2.in‚3.1875 disk.dm-0.in‚ 1116.9140625 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.nvme0n1.in‚ 618.11328125?‹““‚%‚(:ˆ †ÐêÔ ‘¨ÿ gpu.0.gpu‚41.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚169.942' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.4855 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Зbemƒ%‚(:â †ÐêÔÖåÜÿ$ disk./.usageGB‚2712.223846435547 disk.sda1.in‚0 disk.sda1.out‚0! disk.sda2.out‚1155.2666015625 network.recv‚ 3.225633e+06 disk.rbd0.out‚ 2.74609375) proc.memory.availableMB‚ 1291.25390625 cpu‚25.21637884574716) disk./.usagePercent‚75.83491390020534 disk.dm-0.in‚ 1116.9140625 disk.sdb.in‚3.1875 disk.sda.in‚ 6.08203125" disk.nvme1n1.out‚ 53279.9921875 disk.sdb1.out‚0 disk.md0p1.out‚ 1113.484375 disk.rbd2.in‚0! disk.dm-0.out‚110357.76171875 disk.sdb.out‚1155.2666015625 disk.rbd0.in‚0$ memory_percent‚92.11881160736084 disk.nvme0n1.in‚ 618.11328125# disk.nvme0n1.out‚57077.76953125 disk.sda.out‚1155.2666015625 disk.nvme1n1.in‚ 498.80078125! proc.memory.rssMB‚ 3001.234375 proc.cpu.threads‚17 disk.rbd2.out‚ 2.74609375 disk.sdb2.in‚3.1875 disk.md0.out‚ 1113.484375 disk.sdb1.in‚0 disk.md0.in‚ 9.26953125 network.sent‚ 2.2345958e+07! disk.sdb2.out‚1155.2666015625 disk.md0p1.in‚ 9.26953125 disk.sda2.in‚ 6.08203125) proc.memory.percent‚18.31808090209961¶B™4„%/ _wandbruntime‚10085  _runtime‚1008531× Ê…%Ä trainer/global_step‚8649 % _timestamp‚1.7885204600300484e+09 _runtime‚10085.500384397 _step‚709 ' lr-AdamW/pg1‚0.00019479902616582103 ' lr-AdamW/pg2‚0.00019479902616582103k4ýBφ%É ' lr-AdamW/pg1‚0.00019479902616582103 ' lr-AdamW/pg2‚0.00019479902616582103 trainer/global_step‚8649 % _timestamp‚1.7885204600300484e+09 _runtime‚10085.500384397 _step‚709ÅBP$4‡%/ _wandbruntime‚10085  _runtime‚10085dWÊˈ%ûO TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 217164642 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚722296769669632 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 396055802 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 217164642 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 401960960 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410816 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 163240043 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007241 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917410816 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9844358683648 % _timestamp‚1.7885204600309744e+09 _runtime‚10085.501378481 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9844358683648 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚712452410985984 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 178865070 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26088 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚722296769669632 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 216276992 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 178865072 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚712452410985984 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26088 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10278299784192 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9843459275776 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214665 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9807665101354 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚722295852258816 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955591168 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚52018550 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚722295852258816 KBDeviceStatsMonitor. "í€6 on_train_batch_start/allocation.large_pool.peak‚3677 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚797084799724544 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 _step‚710 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177047040 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 178865072 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10278083507200 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚706304751595038 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9843459275776 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚52019138 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26090 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚706305666045818 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 trainer/global_step‚8649 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚712452392983040 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 240217088 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 111221495 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚807362859291648 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 396029712 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚712452392983040 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407872 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450780 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚590 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚797084775784448Wý(‰%€P LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450780 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚590 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚797084775784448 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 217164642 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚722296769669632 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 396055802 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 217164642 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 401960960 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410816 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 163240043 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007241 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917410816 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9844358683648 % _timestamp‚1.7885204600309744e+09 _runtime‚10085.501378481 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9844358683648 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚712452410985984 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 178865070 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26088 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚722296769669632 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 216276992 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 178865072 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚712452410985984 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26088 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10278299784192 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9843459275776 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214665 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9807665101354 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚722295852258816 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955591168 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚52018550 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚722295852258816 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚797084799724544 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 _step‚710 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177047040 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 178865072 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10278083507200 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚706304751595038 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9843459275776 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚52019138 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26090 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚706305666045818 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 trainer/global_step‚8649 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚712452392983040 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 240217088 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 111221495 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚807362859291648 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 396029712 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚712452392983040 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407872Æ›1VM4Š%/ _wandbruntime‚10086  _runtime‚10086á*Ô '‹%…N =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚712532041281536 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10279243575296 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1187 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 217215347 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 217189233 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 163258116 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚604 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9845471500288 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177047040 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 trainer/global_step‚8649 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 178884970 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899421184 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚52024851 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10279468227584 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 178884968 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26114 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 396074201 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚722377512781824 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9844572079104 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚696575981038592 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9807876351252 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚797174095005184 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9845471500288 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 396100317 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 163258722 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 178884968 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚797174118945280 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚606 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚696575963999232 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 396074201 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 401960960 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728s®L5b#Œ%ŠN C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9807876351252 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚797174095005184 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9845471500288 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 396100317 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 163258722 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 178884968 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚797174118945280 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚606 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚696575963999232 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 396074201 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 401960960 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚712532041281536 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10279243575296 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1187 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 217215347 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 217189233 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 163258116 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚604 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9845471500288 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177047040 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 trainer/global_step‚8649 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 178884970 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899421184 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚52024851 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10279468227584 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 178884968 MCDeviceSt+.…‹®atsMonitor.on_train_batch_end/allocation.small_pool.current‚26114 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 396074201 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚722377512781824 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9844572079104 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚696575981038592 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0Ç¡á}V4%/ _wandbruntime‚10086  _runtime‚10086!¹ncŽ%Ý % _timestamp‚1.7885204609939952e+09 _step‚712 0loss/FRt1/assignment_loss‚2.1419677941695503 /loss/FBt1/detection_loss‚2.3667292669415474 /loss/FRt3/assignment_loss‚2.564461780918969 0loss/FRt4/assignment_loss‚2.7595260381698608 1loss/SRqqt3/detection_loss‚1.9064404851068621 1loss/SRqqt4/detection_loss‚2.3160995433727902 epoch‚7 _runtime‚10086.464205939 0loss/FBt2/assignment_loss‚0.1533359029495155 0loss/FBt3/assignment_loss‚0.0304780140884759 3loss/FBt4/assignment_loss‚0.0002765671815723181 /loss/FRt4/detection_loss‚3.0206237991650897 0loss/FRt2/assignment_loss‚1.1913687713810655 2loss/SRqqt2/assignment_loss‚0.7810167635771867 1loss/SRqqt3/assignment_loss‚1.120562370441368 /loss/FBt2/detection_loss‚0.5277463170246054 .loss/FBt4/detection_loss‚3.834116280078888 trainer/global_step‚8649 2loss/SRqqt1/assignment_loss‚1.0897711546951905 /loss/FRt1/detection_loss‚1.8366115115315116 /loss/FBt3/detection_loss‚2.3507872760295867 2loss/FBt1/assignment_loss‚0.019128500774968415 1loss/SRqqt2/detection_loss‚0.8109626326301734 2loss/SRqqt4/assignment_loss‚1.4833258553020034 /loss/FRt2/detection_loss‚1.4985418584747392 /loss/FRt3/detection_loss‚2.5452150346504316 1loss/SRqqt1/detection_loss‚1.6879013166576624 &loss/total_loss‚38.036994835313614E×Æ:h%â /loss/FRt3/detection_loss‚2.5452150346504316 1loss/SRqqt1/detection_loss‚1.6879013166576624 &loss/total_loss‚38.036994835313614 % _timestamp‚1.7885204609939952e+09 _step‚712 0loss/FRt1/assignment_loss‚2.1419677941695503 /loss/FBt1/detection_loss‚2.3667292669415474 /loss/FRt3/assignment_loss‚2.564461780918969 0loss/FRt4/assignment_loss‚2.7595260381698608 1loss/SRqqt3/detection_loss‚1.9064404851068621 1loss/SRqqt4/detection_loss‚2.3160995433727902 epoch‚7 _runtime‚10086.464205939 0loss/FBt2/assignment_loss‚0.1533359029495155 0loss/FBt3/assignment_loss‚0.0304780140884759 3loss/FBt4/assignment_loss‚0.0002765671815723181 /loss/FRt4/detection_loss‚3.0206237991650897 0loss/FRt2/assignment_loss‚1.1913687713810655 2loss/SRqqt2/assignment_loss‚0.7810167635771867 1loss/SRqqt3/assignment_loss‚1.120562370441368 /loss/FBt2/detection_loss‚0.5277463170246054 .loss/FBt4/detection_loss‚3.834116280078888 trainer/global_step‚8649 2loss/SRqqt1/assignment_loss‚1.0897711546951905 /loss/FRt1/detection_loss‚1.8366115115315116 /loss/FBt3/detection_loss‚2.3507872760295867 2loss/FBt1/assignment_loss‚0.019128500774968415 1loss/SRqqt2/detection_loss‚0.8109626326301734 2loss/SRqqt4/assignment_loss‚1.4833258553020034 /loss/FRt2/detection_loss‚1.4985418584747392È~êl”%‚(:‰ •ÐêÔŸþÀ gpu.0.gpu‚37.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚36.0 gpu.0.powerWatts‚114.131' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.53275 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 7bFh‘%‚(:Ý •ÐêÔî·‰ƒ) proc.memory.percent‚18.31808090209961 disk.md0p1.out‚ 1115.5078125 disk.nvme0n1.in‚ 618.11328125# disk.nvme0n1.out‚57077.93359375% disk./.usageGB‚2712.3144187927246! disk.dm-0.out‚110449.96484375 disk.md0p1.in‚ 9.26953125# proc.memory.availableMB‚1291.75 cpu‚25.216040686460648 disk.nvme1n1.in‚ 498.80078125 network.sent‚ 2.2349054e+07 disk.sda1.in‚0 disk.dm-0.in‚ 1116.9140625 disk.sda2.in‚ 6.08203125 disk.sdb1.out‚0 disk.rbd2.in‚0 network.recv‚ 3.229114e+06 disk.sdb2.in‚3.1875! disk.sda2.out‚1157.3583984375 disk.md0.in‚ 9.26953125 proc.cpu.threads‚17 disk.sdb1.in‚0 disk.md0.out‚ 1115.5078125 disk.sdb.in‚3.1875 disk.sdb.out‚1157.3583984375 disk.rbd0.in‚0 disk.rbd2.out‚ 2.74609375! proc.memory.rssMB‚ 3001.234375( disk./.usagePercent‚75.8374463412195! disk.sdb2.out‚1157.3583984375 disk.rbd0.out‚ 2.74609375$ memory_percent‚92.11578369140625 disk.sda1.out‚0 disk.sda.in‚ 6.08203125 disk.sda.out‚1157.3583984375 disk.nvme1n1.out‚ 53372.03125òf‰Ã”’%‚(:‰ ¤ÐêÔó¶Òÿ gpu.0.gpu‚40.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚171.777' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.94425 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ñ+C_“%‚(:Ô ¤ÐêÔ¿Àˆ€ disk.dm-0.out‚110490.8828125 disk.rbd0.in‚0 network.recv‚ 3.234935e+06 disk.nvme0n1.out‚57078.25 disk.sdb.in‚3.1875 disk.sda.in‚ 6.08203125 disk.nvme1n1.in‚ 498.80078125 disk.sdb1.in‚0 disk.rbd2.in‚0 disk.sda1.out‚0 disk.dm-0.in‚ 1116.9140625! proc.memory.rssMB‚ 3001.234375 cpu‚25.215711937071134 disk.nvme0n1.in‚ 618.11328125 disk.sdb.out‚1158.841796875 disk.md0.in‚ 9.26953125 disk.sdb2.out‚1158.841796875) proc.memory.percent‚18.31808090209961% disk./.usageGB‚2712.2891006469727 disk.sda2.in‚ 6.08203125" disk.nvme1n1.out‚ 53412.6328125$ memory_percent‚92.11618900299072 proc.cpu.threads‚17 disk.sda2.out‚1158.841796875 network.sent‚ 2.2458865e+07( disk./.usagePercent‚75.8367384352678 disk.md0.out‚ 1116.9375 disk.rbd2.out‚ 2.74609375 disk.md0p1.out‚ 1116.9375 disk.sdb1.out‚0 disk.rbd0.out‚ 2.74609375 disk.sdb2.in‚3.1875) proc.memory.availableMB‚ 1291.68359375 disk.sda.out‚1158.841796875 disk.sda1.in‚0 disk.md0p1.in‚ 9.269531251ŠâÍ””%‚(:‰ ³ÐêÔÜ´ªþ gpu.0.gpu‚49.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚168.051' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.01275 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0!À•l•%‚(:á ³ÐêÔð¬Ñÿ disk.dm-0.out‚110634.8515625 network.recv‚ 3.238914e+06 disk.rbd0.in‚0 disk.md0.out‚ 1118.34375! proc.memory.rssMB‚ 3001.234375) proc.memory.percent‚18.31808090209961 network.sent‚ 2.2462508e+07! disk.sda2.out‚1160.3056640625 disk.sda1.out‚0 disk.sdb.in‚3.1875) proc.memory.availableMB‚ 1291.44140625 proc.cpu.threads‚17 disk.rbd0.out‚ 2.74609375 disk.md0.in‚ 9.26953125 disk.sda1.in‚0 disk.dm-0.in‚ 1116.9140625 disk.md0p1.in‚ 9.26953125 disk.md0p1.out‚ 1118.34375 cpu‚25.215444729001344 disk.sda.in‚ 6.08203125 disk.sdb1.in‚0 disk.rbd2.in‚0 disk.rbd2.out‚ 2.74609375 disk.sdb2.in‚3.1875 disk.sda.out‚1160.3056640625# disk.nvme1n1.out‚53556.59765625 disk.nvme0n1.in‚ 618.11328125$ memory_percent‚92.11766719818115) disk./.usagePercent‚75.84593513298947 disk.nvme1n1.in‚ 498.80078125! disk.sdb2.out‚1160.3056640625# disk.nvme0n1.out‚57078.25390625 disk.sdb.out‚1160.3056640625$ disk./.usageGB‚2712.618019104004 disk.sda2.in‚ 6.08203125 disk.sdb1.out‚0¤1-4–%/ _wandbruntime‚10139  _runtime‚10139T*Ê—%Ä ' lr-AdamW/pg2‚0.00018963596444194778 trainer/global_step‚8699 % _timestamp‚1.7885205138970053e+09 _runtime‚10139.367381957 _step‚713 ' lr-AdamW/pg1‚0.00018963596444194778×­»Ï˜%É ' lr-AdamW/pg2‚0.00018963596444194778 trainer/global_step‚8699 % _timestamp‚1.7885205138970053e+09 _runtime‚10139.367381957 _step‚713 ' lr-AdamW/pg1‚0.00018963596444194778ÉCbDø4™%/ _wandbruntime‚10139  _runtime‚10139ïÛF(š%ûO F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 398257514 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚811887239729664 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚726333021616640 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚801550760761344 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 218423632 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 111840293 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955591168 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9900013263872 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 179859972 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚52304127 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 401960960 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9899113856000 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚52303502 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177047040 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 % _timestamp‚1.7885205138981676e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9862214347042 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 179859970 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚625 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚811886993221120 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411420 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚726333021616640 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1212 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 218423632 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚710260117409326 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚700397005650864 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26090 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚716433907760640 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9900013263872 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9863111758462 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007241 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 trainer/global_step‚8699 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 218397544 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚627 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 222568448 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚726333939027456 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9899113856000 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 164144422 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚801550736821248 _step‚714 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177047040 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 % _timestamp‚1.7885205138981676e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9862214347042 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 179859970 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚625 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚811886993221120 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411420 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚726333021616640 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1212 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 218423632 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚710260117409326 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚700397005650864 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26090 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚716433907760640 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9900013263872 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9863111758462 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007241 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 trainer/global_step‚8699 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 218397544 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚627 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 222568448 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚726333939027456 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9899113856000 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 164144422 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚801550736821248 _step‚714 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚726333021616640 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚801550760761344 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 218423632 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 111840293 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955591168 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9900013263872 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 179859972 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚52304127 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 401960960 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9899113856000 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚52303502 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 398328119 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 179879870 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚726414682139648 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚716513556059136 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177047040 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1187 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚645 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 179879870 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31686 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899421184 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 164161834 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 179879868 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9900226659328 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10337400567296 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177047040 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 111852671 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚801640056041984 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚710338291713992 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚716513538056192 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917424128 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚10140.339856026 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 218422135 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 261175296 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚52309165 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9901126080512 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚726414682139648 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚52309808 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 398302003 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26114 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚811977456609280}>ì&' %ŠN SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 179879870 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31686 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899421184 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 164161834 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 179879868 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9900226659328 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10337400567296 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177047040 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 111852671 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚801640056041984 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚710338291713992 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚716513538056192 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917424128 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚10140.339856026 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 218422135 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 261175296 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚52309165 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9901126080512 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚726414682139648 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚52309808 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 398302003 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26114 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚811977456609280 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 398328119 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 179879870 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚726414682139648 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚716513556059136 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177047040 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1187 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚645ËXyvÈ4¡%/ _wandbruntime‚10140  _runtime‚10140#yÞ]¢%× epoch‚7 2loss/SRqqt4/assignment_loss‚1.4856541792551676 /loss/FRt3/detection_loss‚2.3985006724380784 /loss/FBt2/detection_loss‚0.6193891591613245 .loss/FBt4/detection_loss‚2.180344228943189 %loss/total_loss‚38.53864422168704 1loss/FBt4/assignment_loss‚0.08237036380523932 /loss/FRt2/detection_loss‚1.4692177217392648 /loss/FRt1/assignment_loss‚2.322323983762322 2loss/SRqqt2/assignment_loss‚0.6222896488254954 % _timestamp‚1.7885205148711357e+09 _runtime‚10140.341246313 _step‚716 2loss/SRqqt3/assignment_loss‚1.1241330032998866 1loss/SRqqt1/detection_loss‚1.7756521456771426 0loss/FRt4/assignment_loss‚3.3130400757635794 2loss/SRqqt1/assignment_loss‚1.1527703450785742 /loss/FRt1/detection_loss‚2.3610995598599667 1loss/SRqqt2/detection_loss‚0.8209015730468706 /loss/FBt3/detection_loss‚2.6365823268890383 0loss/FRt2/assignment_loss‚1.3290219052893217 /loss/FRt3/assignment_loss‚2.786358952522278 /loss/FRt4/detection_loss‚2.8191548259027543 1loss/SRqqt3/detection_loss‚1.6074369509111752 trainer/global_step‚8699 1loss/FBt3/assignment_loss‚0.18739580111578108 0loss/SRqqt4/detection_loss‚2.551660007238388 -loss/FBt1/detection_loss‚2.50692917406559 0loss/FBt1/assignment_loss‚0.3179606607188816 1loss/FBt2/assignment_loss‚0.06845695637773558à”‚b£%Ü /loss/FRt1/assignment_loss‚2.322323983762322 2loss/SRqqt2/assignment_loss‚0.6222896488254954 % _timestamp‚1.7885205148711357e+09 _runtime‚10140.341246313 _step‚716 2loss/SRqqt3/assignment_loss‚1.1241330032998866 1loss/SRqqt1/detection_loss‚1.7756521456771426 0loss/FRt4/assignment_loss‚3.3130400757635794 2loss/SRqqt1/assignment_loss‚1.1527703450785742 /loss/FRt1/detection_loss‚2.3610995598599667 1loss/SRqqt2/detection_loss‚0.8209015730468706 /loss/FBt3/detection_loss‚2.6365823268890383 0loss/FRt2/assignment_loss‚1.3290219052893217 /loss/FRt3/assignment_loss‚2.786358952522278 /loss/FRt4/detection_loss‚2.8191548259027543 1loss/SRqqt3/detection_loss‚1.6074369509111752 trainer/global_step‚8699 1loss/FBt3/assignment_loss‚0.18739580111578108 0loss/SRqqt4/detection_loss‚2.551660007238388 -loss/FBt1/detection_loss‚2.50692917406559 0loss/FBt1/assignment_loss‚0.3179606607188816 1loss/FBt2/assignment_loss‚0.06845695637773558 epoch‚7 2loss/SRqqt4/assignment_loss‚1.4856541792551676 /loss/FRt3/detection_loss‚2.3985006724380784 /loss/FBt2/detection_loss‚0.6193891591613245 .loss/FBt4/detection_loss‚2.180344228943189 %loss/total_loss‚38.53864422168704 1loss/FBt4/assignment_loss‚0.08237036380523932 /loss/FRt2/detection_loss‚1.4692177217392648ÌɇqÜ’¤%‚(:‡ ÑÐêÔêšÈþ gpu.0.gpu‚9.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚179.246' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚44.8115 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0A4ßÝd¥%‚(:Ù ÑÐêÔ¡³‹ÿ network.sent‚ 2.2564314e+07 disk.dm-0.out‚ 110851.1875 disk.md0p1.out‚ 1122.00390625 disk.rbd0.in‚0 disk.rbd0.out‚ 2.74609375$ memory_percent‚92.11783409118652 disk.sdb2.in‚3.1875 disk.dm-0.in‚ 1116.9140625 disk.sdb.in‚3.1875 disk.sda.out‚1164.095703125 disk.sdb1.in‚0 disk.sdb1.out‚0 network.recv‚ 3.248251e+06$ disk./.usageGB‚2712.872341156006! proc.memory.rssMB‚ 3001.234375 disk.sda2.out‚1164.095703125 disk.rbd2.out‚ 2.74609375 disk.md0p1.in‚ 9.26953125 disk.nvme0n1.in‚ 618.11328125 disk.sda2.in‚ 6.08203125 disk.nvme1n1.in‚ 498.80078125 disk.md0.out‚ 1122.00390625 disk.rbd2.in‚0( disk./.usagePercent‚75.8530460840055 disk.sdb2.out‚1164.095703125 disk.sda1.out‚0" disk.nvme0n1.out‚ 57078.4609375 disk.md0.in‚ 9.26953125 disk.sda1.in‚0( proc.memory.availableMB‚ 1291.4140625) proc.memory.percent‚18.31808090209961 cpu‚25.214861737211 disk.sdb.out‚1164.095703125 disk.sda.in‚ 6.08203125" disk.nvme1n1.out‚ 53772.7265625 proc.cpu.threads‚17:Hd¥¦%‚(:… àÐêÔ“´§þ gpu.0.gpu‚56.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚37.0 gpu.0.powerWatts‚125.88' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚31.47 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ðËf§%‚(:Û àÐêԣ¬ÿ network.recv‚ 3.253913e+06 disk.md0.out‚ 1123.3359375! disk.sdb2.out‚1165.4853515625 disk.dm-0.in‚ 1116.9140625 disk.nvme1n1.in‚ 498.80078125 cpu‚25.214617839539528 network.sent‚ 2.2661634e+07 disk.sdb2.in‚3.1875) proc.memory.percent‚18.31808090209961) disk./.usagePercent‚75.85301504579066 disk.rbd2.in‚0' proc.memory.availableMB‚ 1291.546875 proc.cpu.threads‚17% disk./.usageGB‚2712.8712310791016 disk.rbd2.out‚ 2.74609375$ memory_percent‚92.11702346801758! proc.memory.rssMB‚ 3001.234375 disk.nvme0n1.in‚ 618.11328125! disk.sda2.out‚1165.4853515625 disk.rbd0.in‚0 disk.sda1.out‚0 disk.dm-0.out‚ 111035.75 disk.sda.out‚1165.4853515625 disk.rbd0.out‚ 2.74609375! disk.nvme1n1.out‚ 53957.234375 disk.sdb1.out‚0 disk.md0.in‚ 9.26953125 disk.sda1.in‚0! disk.nvme0n1.out‚ 57078.515625 disk.sdb.out‚1165.4853515625 disk.sda.in‚ 6.08203125 disk.md0p1.in‚ 9.26953125 disk.md0p1.out‚ 1123.3359375 disk.sdb.in‚3.1875 disk.sda2.in‚ 6.08203125 disk.sdb1.in‚0¯ ’¨%‚(:‡ ïÐêÔÿ¦øý gpu.0.gpu‚55.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚39.0 gpu.0.powerWatts‚175.512' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.878 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Y¸ru©%‚(:ê ïÐêÔ„‹íþ disk.rbd0.out‚ 2.74609375) disk./.usagePercent‚75.85476246528839 disk.sdb1.in‚0 disk.rbd2.in‚0" disk.sdb2.out‚1167.62060546875 disk.dm-0.in‚ 1116.9140625# disk.nvme0n1.out‚57078.51953125! disk.sdb.out‚1167.62060546875 disk.sda.in‚ 6.08203125 disk.sdb.in‚3.1875) proc.memory.availableMB‚ 1291.41015625 disk.sda1.in‚0 network.recv‚ 3.258084e+06 disk.sda2.in‚ 6.08203125 disk.nvme1n1.in‚ 498.80078125) proc.memory.percent‚18.31808090209961! disk.sda.out‚1167.62060546875 disk.sdb1.out‚0 disk.sdb2.in‚3.1875 disk.dm-0.out‚111088.3203125 disk.md0p1.out‚ 1125.4140625 cpu‚25.214329599173738 proc.cpu.threads‚17 disk.md0.in‚ 9.26953125! proc.memory.rssMB‚ 3001.234375 disk.md0p1.in‚ 9.26953125# disk.nvme1n1.out‚54009.80078125$ memory_percent‚92.11785793304443 network.sent‚ 2.2667534e+07 disk.md0.out‚ 1125.4140625" disk.sda2.out‚1167.62060546875% disk./.usageGB‚2712.9337272644043 disk.rbd2.out‚ 2.74609375 disk.sda1.out‚0 disk.nvme0n1.in‚ 618.11328125 disk.rbd0.in‚0üGbí4ª%/ _wandbruntime‚10190  _runtime‚10190Ô…‹›Ê«%Ä trainer/global_step‚8749 % _timestamp‚1.7885205650214415e+09 _runtime‚10190.491879771 _step‚717 ' lr-AdamW/pg1‚0.00018452275462710398 ' lr-AdamW/pg2‚0.00018452275462710398WϸϬ%É ' lr-AdamW/pg1‚0.00018452275462710398 ' lr-AdamW/pg2‚0.00018452275462710398 trainer/global_step‚8749 % _timestamp‚1.7885205650214415e+09 _runtime‚10190.491879771 _step‚717ÍÆ—Iö4­%/ _wandbruntime‚10190  _runtime‚10190“©ƒÔ'®%ûO =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9955667844096 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10394717077504 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 219656534 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚720415422538240 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚806016697858048 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚720415440541184 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177047040 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411420 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955591168 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚720415422538240 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26090 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 219630446 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1212 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚816411438875648 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9918558415570 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450780 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9917661004150 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 219656534 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 219630446 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚621 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 112459093 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214665 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 180854870 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚714214568772834 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955591168 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9954768436224 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7885205650225441e+09 _runtime‚10190.493010307 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚730371108385280 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9955667844096 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9954768436224 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1187 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 248605696 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410816 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 400485316 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 180854872 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚806016721798144 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚730370190974464 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 91741»_>&Ú0816 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 400511406 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10394492411904 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 trainer/global_step‚8749 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚730370190974464 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 165050081 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚816411190269952 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚714213654322054 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 180854872 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚704296010357264 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407872 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407872 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 219630446 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1212 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚816411438875648 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9918558415570 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450780 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9917661004150 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 219656534 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 219630446 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚621 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 112459093 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214665 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 180854870 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚714214568772834 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955591168 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9954768436224 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7885205650225441e+09 _runtime‚10190.493010307 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚730371108385280 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9955667844096 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9954768436224 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1187 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 248605696 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410816 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 400485316 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 180854872 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚806016721798144 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚730370190974464 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917410816 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 400511406 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10394492411904 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 trainer/global_step‚8749 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚730370190974464 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 165050081 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚816411190269952 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚714213654322054 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 180854872 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚704296010357264 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407872 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407872 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177047040 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411420 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955591168 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚720415422538240 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26090ε—¸4°%/ _wandbruntime‚10191  _runtime‚10191Š„˜ '±%…N XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9918769665468 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31686 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 219655037 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 180874770 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 219681151 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 226749440 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177047040 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177047040 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚806106041018880 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9955881239552 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 165067539 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚714293657528488 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450988 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411628 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚720495052833792 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 219681151 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26114 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10395874934784 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 180874768 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 250689536 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 400555921 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚719 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 400529805 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899421184 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚816501915953664 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26114 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚730451851497472 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 401960960 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚714292743077500 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚720495070836736 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 400555921 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917424128 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚730450934073344 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 _runtime‚10191.450702757 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 112471469 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚720495070836736 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899421184 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 400529805 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9919667077096 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 219655037 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 165068161 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 % _timestamp‚1.7885205659796228e+09 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9956780660736 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚719 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 400529805 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899421184 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚816501915953664 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26114 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚730451851497472 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 401960960 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚714292743077500 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚720495070836736 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 400555921 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917424128 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚730450934073344 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 _runtime‚10191.450702757 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 112471469 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚720495070836736 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899421184 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 400529805 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9919667077096 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 219655037 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 165068161 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 % _timestamp‚1.7885205659796228e+09 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9956780660736 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31686 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 219655037 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 180874770 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 219681151 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 226749440 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177047040 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177047040 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚806106041018880 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9955881239552 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 165067539 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚714293657528488 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450988 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411628 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ZFDeviceS^E\ŸRtatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚720495052833792 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 219681151 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26114 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10395874934784 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 180874768 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 250689536 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 400555921 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚734407360332288 _runtime‚10245.233286379 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917410816 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10010423016448 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 244411392 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚718168105685562 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 113077895 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1187 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955591168 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚708195015063664 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407872 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚820935648650752 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10011322424320 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚616 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9973107661258 _step‚722 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚724396937315840 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 181849770 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚810482682834944 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10452965815808 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411420 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚724396937315840 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 135102146652ÏÜ (Ã%„P UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 220889436 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 220863348 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10452965815808 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411420 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚724396937315840 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214665 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9974005072678 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 165956491 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 401960960 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450780 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 220471296 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007241 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚734407360332288 _runtime‚10245.233286379 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917410816 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10010423016448 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 244411392 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚718168105685562 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 113077895 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1187 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955591168 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚708195015063664 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407872 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚820935648650752 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10011322424320 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚616 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9973107661258 _step‚722 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚724396937315840 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 181849770 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚810482682834944 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 402783723 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31686 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 259078144 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10454127392256 H@DevZÿT$!"iceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 165973919 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 402783723 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 220887939 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚52883650 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚734489020855296 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 113090271 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚52884268 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411628 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 220914053 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10011535819776 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 402757607 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 113090269 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 _runtime‚10246.214020043 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚618 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10011535819776 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 402757607 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899421184 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚724476567611392 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 235138048 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007241 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚724476585614336 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚821026129447936 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 trainer/global_step‚8799 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚724476567611392 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450988 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚821025870369792 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚708272978118432 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9975113734204 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9974216322576 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 _step‚723 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712%E8'Æ%ŽN XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚821025870369792 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚708272978118432 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9975113734204 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9974216322576 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 _step‚723 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 % _timestamp‚1.7885206207429624e+09 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10012435240960 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚724476585614336 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899421184 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 402783723 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31686 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 259078144 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10454127392256 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 165973919 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 402783723 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 220887939 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚52883650 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚734489020855296 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 113090271 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚52884268 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411628 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 220914053 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10011535819776 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 402757607 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 113090269 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 _runtime‚10246.214020043 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚618 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10011535819776 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 402757607 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899421184 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚724476567611392 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 235138048 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007241 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚724476585614336 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚821026129447936 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 trainer/global_step‚8799 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚724476567611392 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450988Ó|y¦4Ç%/ _wandbruntime‚10246  _runtime‚10246 lR«\È%Ö /loss/FRt2/detection_loss‚1.6892467384302132 0loss/SRqqt3/detection_loss‚1.324034298972591 /loss/FBt3/detection_loss‚2.4372955759366355 %loss/total_loss‚38.39839170005176 2loss/SRqqt2/assignment_loss‚0.9798678231816138 /loss/FBt2/detection_loss‚0.4608756708003944 2loss/SRqqt4/assignment_loss‚1.8665615695494193 1loss/SRqqt4/detection_loss‚2.8877792347360542 .loss/FBt4/detection_loss‚2.260557540825435 trainer/global_step‚8799 0loss/FRt3/assignment_loss‚2.4624177801112332 1loss/FBt1/assignment_loss‚0.20299603257444687 0loss/SRqqt1/detection_loss‚1.405349070664312 _step‚724 1loss/FBt2/assignment_loss‚0.12511608637167523 -loss/FRt3/detection_loss‚2.24516759502391 1loss/SRqqt2/detection_loss‚1.1118901275218493 2loss/SRqqt3/assignment_loss‚1.3357019092401732 /loss/FBt1/detection_loss‚2.5197259187698364 0loss/FRt1/assignment_loss‚2.4458558571338656 0loss/FRt4/assignment_loss‚3.2639961426074686 2loss/SRqqt1/assignment_loss‚1.0464471310651937 /loss/FRt1/detection_loss‚1.9097756831756942 .loss/FRt4/detection_loss‚2.994692781796822 epoch‚7 /loss/FRt2/assignment_loss‚1.277683035747425 1loss/FBt3/assignment_loss‚0.03191175103024256 1loss/FBt4/assignment_loss‚0.11344634478525352 % _timestamp‚1.7885206207454073e+09 _runtime‚10246.21552722¹M:aÉ%Û 2loss/SRqqt3/assignment_loss‚1.3357019092401732 /loss/FBt1/detection_loss‚2.5197259187698364 0loss/FRt1/assignment_loss‚2.4458558571338656 0loss/FRt4/assignment_loss‚3.2639961426074686 2loss/SRqqt1/assignment_loss‚1.0464471310651937 /loss/FRt1/detection_loss‚1.9097756831756942 .loss/FRt4/detection_loss‚2.994692781796822 epoch‚7 /loss/FRt2/assignment_loss‚1.277683035747425 1loss/FBt3/assignment_loss‚0.03191175103024256 1loss/FBt4/assignment_loss‚0.11344634478525352 % _timestamp‚1.7885206207454073e+09 _runtime‚10246.21552722 /loss/FRt2/detection_loss‚1.6892467384302132 0loss/SRqqt3/detection_loss‚1.324034298972591 /loss/FBt3/detection_loss‚2.4372955759366355 %loss/total_loss‚38.39839170005176 2loss/SRqqt2/assignment_loss‚0.9798678231816138 /loss/FBt2/detection_loss‚0.4608756708003944 2loss/SRqqt4/assignment_loss‚1.8665615695494193 1loss/SRqqt4/detection_loss‚2.8877792347360542 .loss/FBt4/detection_loss‚2.260557540825435 trainer/global_step‚8799 0loss/FRt3/assignment_loss‚2.4624177801112332 1loss/FBt1/assignment_loss‚0.20299603257444687 0loss/SRqqt1/detection_loss‚1.405349070664312 _step‚724 1loss/FBt2/assignment_loss‚0.12511608637167523 -loss/FRt3/detection_loss‚2.24516759502391 1loss/SRqqt2/detection_loss‚1.1118901275218493Ô› ØÝ“Ê%‚(:ˆ ºÑêÔ°ƒÙÿ gpu.0.gpu‚31.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚172.118' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.0295 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0PÇBtË%‚(:é ºÑêÔΈ‰! disk.sda.out‚1175.99951171875! proc.memory.rssMB‚ 3001.234375 disk.nvme0n1.in‚ 618.11328125 disk.rbd0.in‚0 disk.rbd0.out‚ 2.74609375 disk.sda.in‚ 6.08203125# disk.nvme1n1.out‚54844.44140625 network.recv‚ 3.280844e+06 disk.sda1.in‚0 disk.dm-0.in‚ 1116.9140625! disk.dm-0.out‚111923.29296875" disk.nvme0n1.out‚ 57078.8515625 disk.sdb.in‚3.1875 disk.rbd2.in‚0 disk.sdb2.in‚3.1875" disk.sdb2.out‚1175.99951171875 disk.sda1.out‚0! disk.sdb.out‚1175.99951171875 disk.rbd2.out‚ 2.74609375 disk.nvme1n1.in‚ 498.80078125$ memory_percent‚92.12002754211426" disk.sda2.out‚1175.99951171875) proc.memory.percent‚18.31808090209961 disk.sda2.in‚ 6.08203125 disk.md0.out‚ 1133.52734375 disk.md0p1.out‚ 1133.52734375 disk.sdb1.in‚0 disk.sdb1.out‚0( proc.memory.availableMB‚ 1291.0546875 cpu‚25.21304701367443 proc.cpu.threads‚17 network.sent‚ 2.2811937e+07$ disk./.usageGB‚2713.370693206787 disk.md0p1.in‚ 9.26953125) disk./.usagePercent‚75.86698021592187 disk.md0.in‚ 9.26953125JŠ?N”Ì%‚(:‰ ÉÑêԧʷÿ gpu.0.gpu‚36.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚115.025' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.75625 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÃbkÝmÍ%‚(:â ÉÑêÔ¶÷°€ disk.sda2.in‚ 6.08203125 network.recv‚ 3.28457e+06 disk.md0.out‚ 1134.9296875 disk.sdb.out‚1177.4697265625 disk.sda.in‚ 6.08203125 disk.sda.out‚1177.4697265625( proc.memory.availableMB‚ 1291.0546875 disk.sdb1.out‚0 disk.dm-0.in‚ 1116.9140625# disk.nvme1n1.out‚55050.75390625$ memory_percent‚92.12002754211426 cpu‚25.212788901488544# disk.nvme0n1.out‚57078.85546875! disk.sda2.out‚1177.4697265625 disk.rbd2.in‚0 disk.sdb2.in‚3.1875! disk.sdb2.out‚1177.4697265625 disk.sda1.out‚0 disk.md0p1.in‚ 9.26953125 disk.md0p1.out‚ 1134.9296875 disk.sdb.in‚3.1875 disk.nvme1n1.in‚ 498.80078125 network.sent‚ 2.290509e+07% disk./.usageGB‚2713.3714141845703 disk.nvme0n1.in‚ 618.11328125) proc.memory.percent‚18.31808090209961 proc.cpu.threads‚17) disk./.usagePercent‚75.86700037476245 disk.rbd2.out‚ 2.74609375 disk.dm-0.out‚ 112129.609375 disk.rbd0.out‚ 2.74609375! proc.memory.rssMB‚ 3001.234375 disk.sdb1.in‚0 disk.md0.in‚ 9.26953125 disk.sda1.in‚0 disk.rbd0.in‚09 ¼Ø”Î%‚(:‰ ØÑêÔæ¶àý gpu.0.gpu‚34.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚164.039' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚41.00975 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚03¢Í]mÏ%‚(:â ØÑêÔÜ×ÿ disk.nvme0n1.in‚ 618.11328125 disk.rbd0.out‚ 2.74609375 disk.nvme1n1.out‚ 55260.78125$ memory_percent‚92.11993217468262 proc.cpu.threads‚17 disk.md0.in‚ 9.26953125 disk.md0p1.in‚ 9.26953125 disk.md0p1.out‚ 1137.0546875 disk.sdb.in‚3.1875 disk.rbd0.in‚0( proc.memory.availableMB‚ 1291.0703125 disk.rbd2.out‚ 2.74609375) proc.memory.percent‚18.31808090209961 cpu‚25.212531477703223 disk.rbd2.in‚0 disk.sdb2.in‚3.1875 disk.sdb.out‚1179.6474609375 disk.sda.in‚ 6.08203125 disk.sda.out‚1179.6474609375 disk.sda2.in‚ 6.08203125 disk.sdb1.in‚0! proc.memory.rssMB‚ 3001.234375 disk.md0.out‚ 1137.0546875 disk.sdb1.out‚0! disk.sdb2.out‚1179.6474609375 disk.nvme1n1.in‚ 498.80078125 network.recv‚ 3.289082e+06$ disk./.usageGB‚2713.496810913086 disk.sda1.in‚0 disk.sda1.out‚0! disk.dm-0.out‚112339.73828125# disk.nvme0n1.out‚57078.95703125 network.sent‚ 2.2911035e+07) disk./.usagePercent‚75.87050651977435 disk.dm-0.in‚ 1116.9140625! disk.sda2.out‚1179.6474609375¡m†U4Ð%/ _wandbruntime‚10301  _runtime‚10301ËйSÊÑ%Ä ' lr-AdamW/pg1‚0.00017445014245442183 ' lr-AdamW/pg2‚0.00017445014245442183 trainer/global_step‚8849 % _timestamp‚1.7885206763372793e+09 _runtime‚10301.807880424 _step‚725çÊ?ÝÏÒ%É _runtime‚10301.807880424 _step‚725 ' lr-AdamW/pg1‚0.00017445014245442183 ' lr-AdamW/pg2‚0.00017445014245442183 trainer/global_step‚8849 % _timestamp‚1.7885206763372793e+09ÕU´Ð4Ó%/ _wandbruntime‚10301  _runtime‚10301—V•H`Ô%€P G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚825459637064704 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 404940920 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 182844670 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26088 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410816 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955591168 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _runtime‚10301.809616402 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 222096250 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 166862585 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚814948643871744 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚712094002730704 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411420 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917410816 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407872 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10510993192960 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 401960960 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚722122557049070 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚738445447100928 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10029451729786 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26090 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10028554318366 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚728378470096384 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 _step‚726 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚728378452093440 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10510768527360 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚712094019770064 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚738444529690112 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10066977004544 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 248605696 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007241 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10066077596672 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1212 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 113696695 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 222122338 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚825459388459008 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171‘û¿ (Õ%…P M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955591168 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _runtime‚10301.809616402 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 222096250 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 166862585 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚814948643871744 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚712094002730704 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411420 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917410816 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407872 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10510993192960 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 401960960 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚722122557049070 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚738445447100928 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10029451729786 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26090 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10028554318366 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚728378470096384 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 _step‚726 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚728378452093440 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10510768527360 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚712094019770064 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚738444529690112 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10066977004544 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 248605696 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007241 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10066077596672 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1212 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 113696695 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 222122338 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚825459388459008 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚825459637064704 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 404940920 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 182844670 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26088 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410816 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096ÖQ„f4Ö%/ _wandbruntime‚10302  _runtime‚10302bSW6'×%‹N I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26114 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚825550123624960 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26114 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚614 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899421184 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10068089821184 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚722202560255504 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899421184 % _timestamp‚1.7885206773755324e+09 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411628 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10029662979684 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚616 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177047040 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 405011525 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 222146955 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚712171999864192 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚712171982824832 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 182864568 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31686 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007241 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 _runtime‚10302.846515081 _step‚727 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚722201645804516 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 trainer/global_step‚8849 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚53171053 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450988 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚728458100391936 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 166880122 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 182864568 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955591168 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚738526190213120 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 222120841 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚53171667 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚815037963092480 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 405011525 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 404985409 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10067190400000 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚815037939152384 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 222120841 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10068089821184 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917424128 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 404985409 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 166880738 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1187 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚728458082388992 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 182864570 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚825549883421184 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 222146955 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10067190400000 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 216263680 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚738525272788992 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 113709071 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899421184 % _timestamp‚1.7885206773755324e+09 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411628 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10029662979684 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚616 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177047040 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 405011525 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 222146955 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚712171999864192 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚712171982824832 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 182864568 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31686 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007241 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 _runtime‚10302.846515081 _step‚727 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚722201645804516 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 trainer/global_step‚8849 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚53171053 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450988 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚728458100391936 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 166880122 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 182864568 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955591168 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚738526190213120 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 222120841 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚53171667 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚815037963092480 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 405011525 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 404985409 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10067190400000 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚815037939152384 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 222120841 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10068089821184 OADeviceStatsMonitor.on_trainyÎÏ/_batch_end/allocated_bytes.all.current‚ 917424128 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 404985409 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 166880738 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1187 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚728458082388992 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 182864570 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚825549883421184 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 222146955 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10067190400000 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 216263680 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚738525272788992 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 113709071 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚732359984873984 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚829983699523584 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _runtime‚10356.909656804 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10084000975474 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 223355240 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 114315493 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚53453997 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917410816 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10122631584768 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚732359966871040 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10121732176896 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚742481699047936 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 223329152 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1212 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚819414580968448 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450780 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚715993024476464 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26090 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410816 trainer/global_step‚8899 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 183839572 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 167768883 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10569118555136 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10084898386894 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411420 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955591168 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 183839572 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26088 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 167769492 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 114315495 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚819414604908544 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚742482616458752 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 183839570 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚53453390 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1187 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407872 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚742481699047936 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚726077008412578 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚609 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407872 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 407194812 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2öý p (é%…P ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 407168722 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚742482616458752 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007241 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 223329152 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚732359984873984 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚829983699523584 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _runtime‚10356.909656804 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10084000975474 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 223355240 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 114315493 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚53453997 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917410816 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10122631584768 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚732359966871040 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10121732176896 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚742481699047936 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 223329152 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1212 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚819414580968448 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450780 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚715993024476464 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26090 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410816 trainer/global_step‚8899 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 183839572 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 167768883 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10569118555136 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10084898386894 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411420 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955591168 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 183839572 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26088 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 167769492 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 114315495 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚819414604908544 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚742482616458752 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 183839570 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚53453390 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1187 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407872 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚742481699047936 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚726077008412578 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚609 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407872 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 407194812 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214665 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚732359966871040 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 212082688 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚726077922863358 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10569330637824 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚732359984873984 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 $ _timestamp‚1.788520731439253e+09 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 223355240 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10121732176896 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 JDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚716070987531232 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚732439615169536 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10086007048420 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚716071004570592 trainer/global_step‚8899 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26114 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚742563359570944 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚819503900189184 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10122844980224 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 183859470 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 183859468 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 240203776 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 _step‚731 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚732439597166592 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10570287429632 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚726156097168024 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177047040 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 223379857 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10570503693312 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _runtime‚10357.847101022 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 183859468 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚53459701 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955591168 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚53459112 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899421184 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 223379857 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 114327869 % _timestamp‚1.7885207323762975e+09 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚742562442146816 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899421184 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 167786981 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007241 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚819503924129280 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚726157011619012 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 407239327 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 407213211 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 407239327 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10123744401408 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 167787572 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 216263680 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 183859470 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26114 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚732439615169536 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10122844980224 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 223353743 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚591 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 114327871 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚589 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚830074187618816 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411628 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚742563359570944 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10085109636792 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917424128 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 223353743 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31686 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1187 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0ùg‘'ì%N I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26114 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚742563359570944 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚819503900189184 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10122844980224 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 183859470 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 183859468 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 240203776 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 _step‚731 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚732439597166592 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10570287429632 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚726156097168024 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177047040 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 223379857 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10570503693312 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _runtime‚10357.847101022 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 183859468 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚53459701 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955591168 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚53459112 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899421184 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 223379857 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 114327869 % _timestamp‚1.7885207323762975e+09 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚742562442146816 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899421184 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 167786981 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007241 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚819503924129280 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚726157011619012 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 407239327 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 407213211 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 407239327 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10123744401408 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 167787572 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 216263680 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 183859470 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26114 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚732439615169536 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10122844980224 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 223353743 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚591 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 114327871 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚589 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚830074187618816 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411628 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚742563359570944 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10085109636792 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917424128 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 223353743 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31686 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1187 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚716070987531232 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚732439615169536 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10086007048420 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚716071004570592 trainer/global_step‚8899ÛZ~_ñ4í%/ _wandbruntime‚10357  _runtime‚10357®\&\î%Ö .loss/FBt3/detection_loss‚2.063121646642685 %loss/total_loss‚36.10522438487979 % _timestamp‚1.7885207323784678e+09 /loss/FRt3/detection_loss‚2.4256500150912848 1loss/SRqqt1/detection_loss‚1.3547113271237448 1loss/SRqqt2/detection_loss‚1.0325143427277605 _runtime‚10357.848580699 _step‚732 /loss/FRt3/assignment_loss‚2.030316929022471 /loss/FRt4/assignment_loss‚2.199431582291921 epoch‚7 0loss/SRqqt3/detection_loss‚1.341804145160934 /loss/FRt1/assignment_loss‚2.634035643541588 2loss/SRqqt2/assignment_loss‚1.1557546753643289 1loss/FBt3/assignment_loss‚0.17934090643611852 0loss/FRt2/assignment_loss‚1.2358827287075567 2loss/SRqqt3/assignment_loss‚1.2278041589427098 2loss/SRqqt4/assignment_loss‚1.8518615604511328 1loss/FBt2/assignment_loss‚0.13619279154580882 /loss/FRt1/detection_loss‚1.8784199570764717 /loss/FBt4/detection_loss‚2.2818600101904436 2loss/SRqqt1/assignment_loss‚1.1402818248446644 0loss/FBt4/assignment_loss‚0.2867294986394468 .loss/FRt4/detection_loss‚2.884370187918345 .loss/FBt1/detection_loss‚2.238242618739605 /loss/FBt2/detection_loss‚0.6543132268707268 1loss/FBt1/assignment_loss‚0.23142669266826488 1loss/SRqqt4/detection_loss‚2.1932281809193745 trainer/global_step‚8899 /loss/FRt2/detection_loss‚1.4479297339624049i*Ebaï%Û 0loss/FRt2/assignment_loss‚1.2358827287075567 2loss/SRqqt3/assignment_loss‚1.2278041589427098 2loss/SRqqt4/assignment_loss‚1.8518615604511328 1loss/FBt2/assignment_loss‚0.13619279154580882 /loss/FRt1/detection_loss‚1.8784199570764717 /loss/FBt4/detection_loss‚2.2818600101904436 2loss/SRqqt1/assignment_loss‚1.1402818248446644 0loss/FBt4/assignment_loss‚0.2867294986394468 .loss/FRt4/detection_loss‚2.884370187918345 .loss/FBt1/detection_loss‚2.238242618739605 /loss/FBt2/detection_loss‚0.6543132268707268 1loss/FBt1/assignment_loss‚0.23142669266826488 1loss/SRqqt4/detection_loss‚2.1932281809193745 trainer/global_step‚8899 /loss/FRt2/detection_loss‚1.4479297339624049 .loss/FBt3/detection_loss‚2.063121646642685 %loss/total_loss‚36.10522438487979 % _timestamp‚1.7885207323784678e+09 /loss/FRt3/detection_loss‚2.4256500150912848 1loss/SRqqt1/detection_loss‚1.3547113271237448 1loss/SRqqt2/detection_loss‚1.0325143427277605 _runtime‚10357.848580699 _step‚732 /loss/FRt3/assignment_loss‚2.030316929022471 /loss/FRt4/assignment_loss‚2.199431582291921 epoch‚7 0loss/SRqqt3/detection_loss‚1.341804145160934 /loss/FRt1/assignment_loss‚2.634035643541588 2loss/SRqqt2/assignment_loss‚1.1557546753643289 1loss/FBt3/assignment_loss‚0.17934090643611852Ü•¾’ð%‚(:‡ £ÒêÔÞáØÿ gpu.0.gpu‚57.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚39.0 gpu.0.powerWatts‚164.372' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚41.093 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0á_e tñ%‚(:é £ÒêÔî˜Å€ disk.sdb.in‚3.1875" disk.sda2.out‚1188.04833984375# disk.nvme1n1.out‚56361.55078125 disk.rbd2.in‚0) disk./.usagePercent‚75.85151539870901 disk.rbd0.out‚ 2.74609375$ memory_percent‚92.11981296539307 proc.cpu.threads‚17 disk.sda1.in‚0 disk.sda1.out‚0! disk.sda.out‚1188.04833984375 disk.sda2.in‚ 6.08203125) proc.memory.availableMB‚ 1291.08984375) proc.memory.percent‚18.31808090209961 disk.sdb2.in‚3.1875 network.sent‚ 2.3055004e+07! disk.sdb.out‚1188.04833984375 disk.sda.in‚ 6.08203125 network.recv‚ 3.31121e+06$ disk./.usageGB‚2712.817596435547 disk.nvme0n1.in‚ 618.11328125# disk.nvme0n1.out‚57079.67578125 disk.md0.in‚ 9.26953125 disk.dm-0.out‚113441.2265625! proc.memory.rssMB‚ 3001.234375 disk.rbd0.in‚0 disk.nvme1n1.in‚ 498.80078125 disk.md0.out‚ 1145.18359375 disk.dm-0.in‚ 1116.9140625 cpu‚25.21118547958159 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.rbd2.out‚ 2.74609375" disk.sdb2.out‚1188.04833984375 disk.md0p1.in‚ 9.26953125 disk.md0p1.out‚ 1145.18359375ú›½æ“ò%‚(:ˆ ²ÒêÔ‚Šùý gpu.0.gpu‚29.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚160.922' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚40.2305 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0•Ô;æpó%‚(:å ²ÒêÔ€Œçƒ disk.md0.out‚ 1146.56640625 disk.sdb2.in‚3.1875 disk.dm-0.in‚ 1116.9140625 disk.nvme0n1.in‚ 618.11328125 disk.sdb1.in‚0 disk.sdb1.out‚0 network.sent‚ 2.3152339e+07 disk.md0p1.in‚ 9.26953125) proc.memory.percent‚18.31808090209961$ memory_percent‚92.12126731872559 proc.cpu.threads‚17! disk.dm-0.out‚113513.26953125 disk.sdb.in‚3.1875 disk.rbd0.in‚0 disk.rbd2.in‚0 disk.md0.in‚ 9.26953125 disk.sda.out‚1189.4873046875! disk.sda2.out‚1189.4873046875# disk.nvme1n1.out‚56433.48046875 disk.rbd2.out‚ 2.74609375 cpu‚25.21084055084799 network.recv‚ 3.317277e+06) disk./.usagePercent‚75.85151347881943$ disk./.usageGB‚2712.817527770996! disk.sdb2.out‚1189.4873046875 disk.sda1.in‚0" disk.nvme0n1.out‚ 57079.7890625 disk.rbd0.out‚ 2.74609375! proc.memory.rssMB‚ 3001.234375 disk.sda1.out‚0 disk.md0p1.out‚ 1146.56640625 disk.sda2.in‚ 6.08203125( proc.memory.availableMB‚ 1290.8515625 disk.sdb.out‚1189.4873046875 disk.sda.in‚ 6.08203125 disk.nvme1n1.in‚ 498.80078125”ö‘ô%‚(:† ÁÒêÔ÷ßÿ gpu.0.gpu‚57.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚37.0 gpu.0.powerWatts‚114.74' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.685 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0óƵ$mõ%‚(:â ÁÒêÔ°Ö÷ÿ cpu‚25.21053258445563( disk./.usagePercent‚75.8341597035827# disk.nvme0n1.out‚57079.87890625 disk.sdb.in‚3.1875 disk.sda.in‚ 6.08203125 disk.sda2.in‚ 6.08203125 disk.nvme1n1.in‚ 498.80078125 disk.md0.out‚ 1148.60546875 disk.sdb2.in‚3.1875 disk.md0.in‚ 9.26953125 disk.sda1.in‚0& proc.memory.availableMB‚ 1291.09375 disk.nvme0n1.in‚ 619.5859375 disk.rbd0.out‚ 2.74609375 disk.sdb1.in‚0! disk.sdb2.out‚1191.5849609375$ memory_percent‚92.11978912353516 disk.rbd2.out‚ 2.74609375 disk.dm-0.in‚ 1118.38671875 disk.md0p1.out‚ 1148.60546875! disk.sda2.out‚1191.5849609375 disk.sdb1.out‚0 proc.cpu.threads‚17 disk.rbd2.in‚0 disk.sda1.out‚0! proc.memory.rssMB‚ 3001.234375 network.sent‚ 2.3157601e+07 disk.md0p1.in‚ 9.26953125! disk.nvme1n1.out‚ 56603.078125) proc.memory.percent‚18.31808090209961 disk.sda.out‚1191.5849609375 network.recv‚ 3.320521e+06% disk./.usageGB‚2712.1968727111816! disk.dm-0.out‚113682.95703125 disk.sdb.out‚1191.5849609375 disk.rbd0.in‚0‘B¤÷4ö%/ _wandbruntime‚10409  _runtime‚10409ý£‚É÷%à ' lr-AdamW/pg1‚0.00016458958567944036 ' lr-AdamW/pg2‚0.00016458958567944036 trainer/global_step‚8949 % _timestamp‚1.7885207842162735e+09 _runtime‚10409.68684098 _step‚733ôKv¯Îø%È ' lr-AdamW/pg1‚0.00016458958567944036 ' lr-AdamW/pg2‚0.00016458958567944036 trainer/global_step‚8949 % _timestamp‚1.7885207842162735e+09 _runtime‚10409.68684098 _step‚733ÝvÍ 4ù%/ _wandbruntime‚10409  _runtime‚10409+øØ{ú%€P \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚746518868405760 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 114934293 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007241 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 409396524 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 224562054 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10627821679104 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10140345044002 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚53742562 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚736341499651584 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 224562054 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚719892012143504 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 184834470 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407872 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 184834470 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 224588142 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.largسJ–‹e_pool.peak‚ 12392107008 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 184834472 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚641 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 244411392 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10627601207808 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 168677498 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 trainer/global_step‚8949 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚736341481648640 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 401960960 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚834508387624448 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚823880542005248 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1187 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1212 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10139447632582 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚834508143213056 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410816 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10177386757120 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚730032374226866 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 409422614 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 220471296 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26088 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955591168 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚730031459776086 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚823880565945344 _step‚734 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10178286164992 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10178286164992 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411420 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 224588142 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚736341481648640 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26088 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955591168 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚643õ·èH (û%…P \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚746518868405760 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 114934293 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007241 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 409396524 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 224562054 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10627821679104 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10140345044002 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚53742562 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚736341499651584 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 224562054 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚719892012143504 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 184834470 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407872 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 184834470 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 224588142 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 184834472 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚641 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 244411392 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10627601207808 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 168677498 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 trainer/global_step‚8949 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚736341481648640 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 401960960 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚834508387624448 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚823880542005248 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1187 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1212 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10139447632582 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚834508143213056 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410816 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10177386757120 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚730032374226866 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 409422614 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 220471296 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26088 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955591168 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚730031459776086 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚823880565945344 _step‚734 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10178286164992 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10178286164992 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411420 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 224588142 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚736341481648640 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26088 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955591168 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚643Þë( ”ü%‚(:‰ ÐÒêÔÕöÉþ gpu.0.gpu‚23.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚168.383' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.09575 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0€ žFhý%‚(:Ý ÐÒêÔäŃÿ$ disk./.usageGB‚2712.337673187256 disk.sda2.in‚ 6.08203125 disk.sdb1.in‚0 network.sent‚ 2.3161376e+07) disk./.usagePercent‚75.83809654382333 disk.md0.in‚ 9.26953125 disk.dm-0.in‚ 1118.38671875# disk.nvme0n1.out‚57080.08984375 disk.sda.in‚ 6.08203125 disk.sda.out‚1192.9794921875$ memory_percent‚92.12079048156738 network.recv‚ 3.324973e+06 disk.md0p1.in‚ 9.26953125! disk.nvme1n1.out‚ 56675.515625( proc.memory.availableMB‚ 1290.9296875! proc.memory.rssMB‚ 3001.234375 proc.cpu.threads‚17 disk.rbd2.in‚0 disk.sda1.out‚0! disk.dm-0.out‚113755.60546875 disk.rbd0.out‚ 2.74609375 disk.sdb1.out‚0) proc.memory.percent‚18.31808090209961 disk.md0.out‚ 1149.9375 disk.md0p1.out‚ 1149.9375 disk.nvme0n1.in‚ 619.5859375 disk.rbd2.out‚ 2.74609375! disk.sdb2.out‚1192.9794921875 disk.sda1.in‚0! disk.sda2.out‚1192.9794921875 disk.nvme1n1.in‚ 498.80078125 disk.sdb2.in‚3.1875 disk.sdb.in‚3.1875 disk.sdb.out‚1192.9794921875 disk.rbd0.in‚0 cpu‚25.210234603945036¦!Ý4þ%/ _wandbruntime‚10410  _runtime‚10410üâ'ÿ%ŠN SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 184854370 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚834598617848832 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177047040 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 409467129 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26114 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚8949 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚719969992237632 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚736421111944192 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10179398981632 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899421184 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411628 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 % _timestamp‚1.7885207853149524e+09 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26114 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚730110548531532 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 _step‚735 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 224586645 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 168695584 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚736421129947136 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚736421111944192 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10628756622848 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 114946671 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚746600528928768 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10628985469440 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007241 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 409441013 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚823969861225984 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10140556293900 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899421184 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 252786688 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚730111462982520 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 184854370 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10141453705528 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10178499560448 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚633 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚719970009276992 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 401960960 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚53748280 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 _runtime‚10410.78635555 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚746599611504640 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955591168 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917424128 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚635 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚736421129947136 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚834598870635520 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177047040 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚746600528928768 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 409467129 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 224612759 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 224586645 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 168694949 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10179398981632 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚823969885166080 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450988 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411628 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 % _timestamp‚1.7885207853149524e+09 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26114 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚730110548531532 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 _step‚735 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 224586645 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 168695584 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚736421129947136 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚736421111944192 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10628756622848 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 114946671 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚746600528928768 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10628985469440 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007241 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 409441013 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚823969861225984 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10140556293900 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899421184 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 252786688 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚730111462982520 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 184854370 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10141453705528 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10178499560448 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚633 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚719970009276992 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 401960960 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚53748280 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 _runtime‚10410.78635555 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚746599611504640 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UEDeviceStatsMonitor.on_traiîÍ£Ðn_batch_end/requested_bytes.large_pool.peak‚ 12392107008 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955591168 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917424128 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚635 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚736421129947136 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚834598870635520 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177047040 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚746600528928768 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 409467129 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 224612759 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 224586645 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 168694949 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10179398981632 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚823969885166080 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450988 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 409467129 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26114 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚8949 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚719969992237632 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171ßÕܦ4&/ _wandbruntime‚10410  _runtime‚10410vI*=]‚&× 2loss/SRqqt4/assignment_loss‚1.4667830398705388 1loss/FBt2/assignment_loss‚0.03886333291443988 .loss/FRt4/detection_loss‚3.082430348946498 /loss/FBt1/detection_loss‚2.2877375036478043 _runtime‚10410.787466285 1loss/SRqqt1/assignment_loss‚1.455963523272011 2loss/SRqqt3/assignment_loss‚1.0118203554810448 /loss/FRt2/detection_loss‚1.3947867455581824 .loss/FBt3/detection_loss‚3.057538549105326 2loss/SRqqt2/assignment_loss‚0.9581211230944944 0loss/FBt1/assignment_loss‚0.2665536460481235 1loss/FBt3/assignment_loss‚0.09478783848135208 1loss/FBt4/assignment_loss‚0.19850299518066095 .loss/FRt1/detection_loss‚2.279765632519277 % _timestamp‚1.7885207853173025e+09 _step‚736 0loss/SRqqt1/detection_loss‚1.607041029383739 0loss/SRqqt4/detection_loss‚2.531343964593751 /loss/FBt4/detection_loss‚2.0390650779008865 trainer/global_step‚8949 0loss/FRt2/assignment_loss‚1.2896247810787624 0loss/FRt3/assignment_loss‚2.1659331830839315 /loss/FRt4/assignment_loss‚3.131162079480978 1loss/SRqqt2/detection_loss‚1.0036749170146515 &loss/total_loss‚37.941087946878184 epoch‚7 /loss/FRt1/assignment_loss‚2.494004398584366 /loss/FRt3/detection_loss‚2.0700399735942483 1loss/SRqqt3/detection_loss‚1.5685928893360224 0loss/FBt2/detection_loss‚0.446951018707097861aëbƒ&Ü 1loss/FBt4/assignment_loss‚0.19850299518066095 .loss/FRt1/detection_loss‚2.279765632519277 % _timestamp‚1.7885207853173025e+09 _step‚736 0loss/SRqqt1/detection_loss‚1.607041029383739 0loss/SRqqt4/detection_loss‚2.531343964593751 /loss/FBt4/detection_loss‚2.0390650779008865 trainer/global_step‚8949 0loss/FRt2/assignment_loss‚1.2896247810787624 0loss/FRt3/assignment_loss‚2.1659331830839315 /loss/FRt4/assignment_loss‚3.131162079480978 1loss/SRqqt2/detection_loss‚1.0036749170146515 &loss/total_loss‚37.941087946878184 epoch‚7 /loss/FRt1/assignment_loss‚2.494004398584366 /loss/FRt3/detection_loss‚2.0700399735942483 1loss/SRqqt3/detection_loss‚1.5685928893360224 0loss/FBt2/detection_loss‚0.44695101870709786 2loss/SRqqt4/assignment_loss‚1.4667830398705388 1loss/FBt2/assignment_loss‚0.03886333291443988 .loss/FRt4/detection_loss‚3.082430348946498 /loss/FBt1/detection_loss‚2.2877375036478043 _runtime‚10410.787466285 1loss/SRqqt1/assignment_loss‚1.455963523272011 2loss/SRqqt3/assignment_loss‚1.0118203554810448 /loss/FRt2/detection_loss‚1.3947867455581824 .loss/FBt3/detection_loss‚3.057538549105326 2loss/SRqqt2/assignment_loss‚0.9581211230944944 0loss/FBt1/assignment_loss‚0.2665536460481235 1loss/FBt3/assignment_loss‚0.09478783848135208àPéZÙ”„&‚(:‰ ßÒêÔéŒÐþ gpu.0.gpu‚43.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚171.501' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.87525 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0+Ä2•c…&‚(:Ø ßÒêÔö”˜ÿ disk.sdb.in‚3.1875 disk.md0p1.in‚ 9.26953125 disk.sda2.out‚ 1194.7265625 disk.md0.in‚ 9.26953125 disk.rbd2.in‚0$ memory_percent‚92.12100505828857! proc.memory.rssMB‚ 3001.234375 network.sent‚ 2.3257728e+07 disk.sda.in‚ 6.08203125 disk.rbd0.out‚ 2.74609375# disk.nvme1n1.out‚56752.78515625 disk.sdb2.in‚3.1875 disk.sda1.out‚0 cpu‚25.209979827877927% disk./.usageGB‚2712.2792739868164 disk.sda2.in‚ 6.08203125 disk.md0.out‚ 1151.6171875 disk.sdb2.out‚ 1194.7265625 disk.rbd0.in‚0 disk.dm-0.in‚ 1118.38671875! disk.dm-0.out‚113832.94140625 network.recv‚ 3.330389e+06 disk.nvme0n1.in‚ 619.5859375 disk.nvme0n1.out‚ 57080.15625 disk.sdb1.in‚0 disk.sdb1.out‚0) proc.memory.percent‚18.31808090209961 proc.cpu.threads‚17 disk.md0p1.out‚ 1151.6171875 disk.sdb.out‚ 1194.7265625 disk.sda.out‚ 1194.7265625 disk.nvme1n1.in‚ 498.80078125 disk.rbd2.out‚ 2.74609375 disk.sda1.in‚0) proc.memory.availableMB‚ 1290.89453125) disk./.usagePercent‚75.83646367773707•\²’†&‚(:‡ îÒêÔÌùàý gpu.0.gpu‚39.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚169.61' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.4025 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0”T%u‡&‚(:ê îÒêÔ¨ôèþ! disk.sda.out‚1196.21826171875" disk.sda2.out‚1196.21826171875 disk.md0.in‚ 9.26953125 disk.sda1.in‚0 network.sent‚ 2.3355126e+07 disk.rbd2.in‚0 disk.rbd2.out‚ 2.74609375 disk.sdb2.in‚3.1875 disk.md0p1.out‚ 1153.05078125! disk.sdb.out‚1196.21826171875) proc.memory.availableMB‚ 1290.78515625! proc.memory.rssMB‚ 3001.234375 disk.dm-0.in‚ 1118.38671875! disk.dm-0.out‚113925.58203125 disk.nvme1n1.in‚ 498.80078125 disk.sdb1.in‚0) disk./.usagePercent‚75.83897041356299 disk.sda2.in‚ 6.08203125) proc.memory.percent‚18.31808090209961 cpu‚25.209704629571824 disk.rbd0.in‚0 disk.sda1.out‚0 disk.nvme0n1.in‚ 619.5859375! disk.nvme1n1.out‚ 56845.015625 disk.sdb.in‚3.1875 proc.cpu.threads‚17 disk.sdb1.out‚0# disk.nvme0n1.out‚57080.56640625 disk.md0.out‚ 1153.05078125" disk.sdb2.out‚1196.21826171875 disk.md0p1.in‚ 9.26953125 network.recv‚ 3.336435e+06$ disk./.usageGB‚2712.368927001953 disk.sda.in‚ 6.08203125 disk.rbd0.out‚ 2.74609375$ memory_percent‚92.12167263031006ù’?E”ˆ&‚(:‰ ýÒêÔ‰‹ž gpu.0.gpu‚55.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚39.0 gpu.0.powerWatts‚171.777' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.94425 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ñïüq‰&‚(:æ ýÒêÔÐ𶄠disk.sda1.out‚0 disk.dm-0.in‚ 1118.38671875! disk.sdb.out‚1198.14794921875 disk.dm-0.out‚113964.7109375) proc.memory.availableMB‚ 1290.89453125 network.sent‚ 2.3362687e+07 disk.md0p1.out‚ 1154.9140625 disk.sdb.in‚3.1875 disk.sda2.in‚ 6.08203125 disk.sdb1.in‚0 disk.md0.out‚ 1154.9140625 disk.sda1.in‚0% disk./.usageGB‚2712.4968605041504! disk.nvme1n1.out‚ 56884.140625 disk.md0p1.in‚ 9.26953125" disk.nvme0n1.out‚ 57080.5703125 disk.rbd2.out‚ 2.74609375 disk.sdb2.in‚3.1875$ memory_percent‚92.12100505828857) proc.memory.percent‚18.31808090209961 cpu‚25.20944798529995) disk./.usagePercent‚75.84254748782875" disk.sda2.out‚1198.14794921875 disk.md0.in‚ 9.26953125 disk.rbd2.in‚0" disk.sdb2.out‚1198.14794921875 proc.cpu.threads‚17 disk.sda.in‚ 6.08203125! proc.memory.rssMB‚ 3001.234375! disk.sda.out‚1198.14794921875 disk.rbd0.in‚0 disk.nvme1n1.in‚ 498.80078125 disk.sdb1.out‚0 network.recv‚ 3.339893e+06 disk.nvme0n1.in‚ 619.5859375 disk.rbd0.out‚ 2.74609375"6Yµ4Š&/ _wandbruntime‚10463  _runtime‚10463ék_‡Ç‹&Á & lr-AdamW/pg1‚0.0001597414036818409 & lr-AdamW/pg2‚0.0001597414036818409 trainer/global_step‚8999 $ _timestamp‚1.788520837951741e+09 _runtime‚10463.422197718 _step‚737»Ìj.ÌŒ&Æ $ _timestamp‚1.788520837951741e+09 _runtime‚10463.422197718 _step‚737 & lr-AdamW/pg1‚0.0001597414036818409 & lr-AdamW/pg2‚0.0001597414036818409 trainer/global_step‚8999áær®4&/ _wandbruntime‚10463  _runtime‚10463Â[‘h(Ž&P RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 225794956 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1212 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955591168 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚750556955174400 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚625 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚750556955174400 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚740323014429184 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 169584021 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 185829372 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10233940745216 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚54031551 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10233940745216 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917410816 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26090 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10685958383616 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚740322996426240 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚54030928 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚828346526982144 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚740323014429184 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚623 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410816 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177047040 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 169584646 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚839032461425664 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 401960960 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450780 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚750556037763584 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 185829372 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10194894289690 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 trainer/global_step‚8999 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚750556037763584 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407872 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 239345664 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10195791701110 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 411650416 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 411624326 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 225794956 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚723791033889264 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 185829370 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407872 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 % _timestamp‚1.7885208379532948e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 185829370 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 225821044 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 115553093 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26088 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚723791016849904Ï'Œˆ (&†P QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 185829370 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407872 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 % _timestamp‚1.7885208379532948e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 185829370 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 225821044 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 115553093 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26088 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚723791016849904 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 225794956 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1212 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955591168 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚750556955174400 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚625 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚750556955174400 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚740323014429184 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 169584021 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 185829372 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10233940745216 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚54031551 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10233940745216 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917410816 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26090 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10685958383616 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚740322996426240 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚54030928 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚828346526982144 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚740323014429184 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚623 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410816 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177047040 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 169584646 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚839032461425664 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 401960960 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450780 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚750556037763584 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 185829372 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10194894289690 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 trainer/global_step‚8999 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚750556037763584 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407872 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 239345664 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10195791701110 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 411650416 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 411624326 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 225794956 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚723791033889264 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360â˜F`ÿ4&/ _wandbruntime‚10464  _runtime‚10464ÞFwÃ.‘&‹N VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899421184 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 115565469 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1187 VHDeviceStatsMonitor.on_train_batch_end/reñPã>ã%quested_bytes.small_pool.current‚ 897411628 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26114 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 169602642 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚828435822262784 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 411694931 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 225819547 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚54037171 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 411668815 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26114 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 185849268 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 225819547 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚839123188513280 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10687342310400 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 115565471 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10235053561856 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 256980992 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899421184 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚723869013983392 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 185849268 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚632 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚750637698286592 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10235053561856 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450988 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955591168 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 233040896 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007241 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 _runtime‚10464.562326941 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 169602010 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚734065914346028 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 411668815 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10687109269504 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚828435846202880 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177047040 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 411694931 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚8999 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚740402626721792 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 169602010 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚734065914346028 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 411668815 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10687109269504 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚828435846202880 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177047040 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 411694931 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚8999 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚740402626721792 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 185849268 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 225819547 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚839123188513280 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10687342310400 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 115565471 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10235053561856 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 256980992 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899421184 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚723869013983392 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 185849268 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚632 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚750637698286592 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10235053561856 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450988 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984ãlTRz4“&/ _wandbruntime‚10464  _runtime‚10464hgs¾`”&Ú 3loss/FBt1/assignment_loss‚0.0034890018602406294 1loss/FBt2/assignment_loss‚0.10851569014927322 0loss/SRqqt3/detection_loss‚1.234193928539753 .loss/FBt1/detection_loss‚2.681738704442978 .loss/FBt4/detection_loss‚2.634475903851645 epoch‚7 % _timestamp‚1.7885208390940936e+09 .loss/FRt3/detection_loss‚2.209495919978335 /loss/FRt3/assignment_loss‚1.868095772606986 /loss/FRt4/assignment_loss‚3.060049729347229 /loss/FRt2/detection_loss‚1.4726978190243245 0loss/FBt2/detection_loss‚0.46268548738635795 trainer/global_step‚8999 0loss/FRt1/assignment_loss‚2.0990556162230822 /loss/FRt2/assignment_loss‚1.801926612854004 2loss/SRqqt3/assignment_loss‚1.2617255091797888 /loss/FRt4/detection_loss‚2.9857802295684817 1loss/SRqqt2/detection_loss‚1.0435966382424036 2loss/SRqqt1/assignment_loss‚1.0931604005143805 2loss/FBt3/assignment_loss‚0.053132306329462156 1loss/SRqqt4/detection_loss‚2.1931747381503763 &loss/total_loss‚37.381178443278486 2loss/SRqqt2/assignment_loss‚0.9188381512028475 1loss/FBt4/assignment_loss‚0.42911966604255475 /loss/FRt1/detection_loss‚2.1600811907536603 _runtime‚10464.56433558 2loss/SRqqt4/assignment_loss‚1.8002170897447145 1loss/SRqqt1/detection_loss‚1.4846719684615983 .loss/FBt3/detection_loss‚2.321260368824005 _step‚740:1‘ie•&ß 2loss/SRqqt4/assignment_loss‚1.8002170897447145 1loss/SRqqt1/detection_loss‚1.4846719684615983 .loss/FBt3/detection_loss‚2.321260368824005 _step‚740 3loss/FBt1/assignment_loss‚0.0034890018602406294 1loss/FBt2/assignment_loss‚0.10851569014927322 0loss/SRqqt3/detection_loss‚1.234193928539753 .loss/FBt1/detection_loss‚2.681738704442978 .loss/FBt4/detection_loss‚2.634475903851645 epoch‚7 % _timestamp‚1.7885208390940936e+09 .loss/FRt3/detection_loss‚2.209495919978335 /loss/FRt3/assignment_loss‚1.868095772606986 /loss/FRt4/assignment_loss‚3.060049729347229 /loss/FRt2/detection_loss‚1.4726978190243245 0loss/FBt2/detection_loss‚0.46268548738635795 trainer/global_step‚8999 0loss/FRt1/assignment_loss‚2.0990556162230822 /loss/FRt2/assignment_loss‚1.801926612854004 2loss/SRqqt3/assignment_loss‚1.2617255091797888 /loss/FRt4/detection_loss‚2.9857802295684817 1loss/SRqqt2/detection_loss‚1.0435966382424036 2loss/SRqqt1/assignment_loss‚1.0931604005143805 2loss/FBt3/assignment_loss‚0.053132306329462156 1loss/SRqqt4/detection_loss‚2.1931747381503763 &loss/total_loss‚37.381178443278486 2loss/SRqqt2/assignment_loss‚0.9188381512028475 1loss/FBt4/assignment_loss‚0.42911966604255475 /loss/FRt1/detection_loss‚2.1600811907536603 _runtime‚10464.56433558ä§Æ‚”–&‚(:‰ ŒÓêÔšÜÜý gpu.0.gpu‚54.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚173.059' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.26475 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¹×TŽj—&‚(:ß ŒÓêÔ°Œ¶€$ disk./.usageGB‚2712.872814178467 disk.sdb2.in‚3.1875 disk.md0p1.out‚1156.25 disk.sda.in‚ 6.08203125 disk.dm-0.out‚114089.6328125! disk.sdb.out‚1199.54833984375# disk.nvme1n1.out‚57008.94921875 disk.md0.out‚1156.25) proc.memory.availableMB‚ 1290.87109375" disk.sda2.out‚1199.54833984375 network.sent‚ 2.3366528e+07) disk./.usagePercent‚75.85305930991147 disk.dm-0.in‚ 1118.38671875 disk.nvme0n1.in‚ 619.5859375 disk.sda2.in‚ 6.08203125 disk.rbd0.in‚0 disk.md0.in‚ 9.26953125 disk.rbd2.in‚0 disk.sda1.in‚0 disk.rbd2.out‚ 2.74609375$ memory_percent‚92.12114810943604! proc.memory.rssMB‚ 3001.234375) proc.memory.percent‚18.31808090209961 cpu‚25.209192462001777 disk.sda1.out‚0# disk.nvme0n1.out‚57080.68359375" disk.sdb2.out‚1199.54833984375 disk.md0p1.in‚ 9.26953125 disk.rbd0.out‚ 2.74609375 disk.nvme1n1.in‚ 498.80078125 disk.sdb1.in‚0 disk.sdb1.out‚0 network.recv‚ 3.344139e+06 disk.sdb.in‚3.1875! disk.sda.out‚1199.54833984375 proc.cpu.threads‚17D’˜&‚(:‡ ›ÓêÔ†î•þ gpu.0.gpu‚45.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚169.61' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.4025 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0t´u‡c™&‚(:Ø ›ÓêÔ‹ß›ÿ disk.md0p1.in‚ 9.26953125( disk./.usagePercent‚75.8600437615364% disk./.usageGB‚2713.1226119995117 disk.sdb1.in‚0 disk.dm-0.in‚ 1118.38671875! proc.memory.rssMB‚ 3001.234375 disk.nvme1n1.in‚ 498.80078125 disk.sda1.out‚0 disk.md0p1.out‚ 1157.62109375 disk.sdb.out‚ 1200.98828125 disk.sda.in‚ 6.08203125 disk.sda.out‚ 1200.98828125 disk.sda2.in‚ 6.08203125 disk.md0.in‚ 9.26953125 disk.md0.out‚ 1157.62109375 proc.cpu.threads‚17 disk.sda2.out‚ 1200.98828125 disk.rbd0.out‚ 2.74609375 disk.nvme0n1.out‚ 57080.6875 network.recv‚ 3.34905e+06 disk.sdb.in‚3.1875 disk.rbd2.in‚0 disk.sdb2.in‚3.1875) proc.memory.percent‚18.31808090209961 disk.nvme0n1.in‚ 619.5859375 disk.rbd0.in‚0" disk.nvme1n1.out‚ 57238.1953125 disk.sdb1.out‚0 disk.rbd2.out‚ 2.74609375 disk.sdb2.out‚ 1200.98828125 disk.sda1.in‚0 disk.dm-0.out‚114318.8828125) proc.memory.availableMB‚ 1290.86328125 network.sent‚ 2.3474294e+07$ memory_percent‚92.12119579315186 cpu‚25.20894747871232Èï8”š&‚(:‰ ªÓêÔ†“öý gpu.0.gpu‚41.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚37.0 gpu.0.powerWatts‚124.321' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚31.08025 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0M@t›&‚(:é ªÓêÔÏשÿ disk.sdb1.out‚0 disk.sdb2.in‚3.1875 disk.dm-0.in‚ 1118.38671875 disk.nvme0n1.in‚ 619.5859375 disk.sdb.in‚3.1875$ disk./.usageGB‚2713.371593475342 disk.sda.in‚ 6.08203125 disk.rbd0.in‚0 disk.md0p1.out‚ 1159.57421875! disk.sda.out‚1203.00244140625 disk.md0.out‚ 1159.57421875" disk.nvme0n1.out‚ 57080.7421875 disk.md0.in‚ 9.26953125 disk.sda1.in‚0 disk.sda1.out‚0 network.sent‚ 2.3480572e+07 disk.sda2.in‚ 6.08203125 disk.md0p1.in‚ 9.26953125$ memory_percent‚92.12117195129395) disk./.usagePercent‚75.86700538780745# disk.nvme1n1.out‚57555.98828125 disk.rbd2.in‚0 network.recv‚ 3.35391e+06 disk.rbd0.out‚ 2.74609375 disk.sdb1.in‚0! disk.dm-0.out‚114636.73046875! disk.sdb.out‚1203.00244140625( proc.memory.availableMB‚ 1290.8671875 proc.cpu.threads‚17 disk.nvme1n1.in‚ 498.80078125 disk.rbd2.out‚ 2.74609375" disk.sdb2.out‚1203.00244140625! proc.memory.rssMB‚ 3001.234375) proc.memory.percent‚18.31808090209961 cpu‚25.208696761764998" disk.sda2.out‚1203.00244140625~À2È“œ&‚(:ˆ ¹ÓêÔŽ †þ gpu.0.gpu‚39.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚167.434' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚41.8585 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Äbrâq&‚(:æ ¹ÓêÔÈ•’ÿ network.recv‚ 3.356592e+06( disk./.usagePercent‚75.8810420204952 disk.rbd0.in‚0 disk.rbd2.in‚0 disk.sdb.in‚3.1875! disk.sdb.out‚1204.45068359375 cpu‚25.208471753370173" disk.sda2.out‚1204.45068359375 disk.md0.in‚ 9.26953125$ memory_percent‚92.12117195129395) proc.memory.percent‚18.31808090209961% disk./.usageGB‚2713.8736114501953 disk.sda.in‚ 6.08203125 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.sdb2.in‚3.1875 disk.sda1.in‚0 disk.dm-0.in‚ 1118.38671875 disk.md0p1.out‚ 1160.9765625 network.sent‚ 2.3484945e+07! disk.sda.out‚1204.45068359375 disk.sda1.out‚0 disk.nvme0n1.in‚ 619.5859375 proc.cpu.threads‚17 disk.nvme1n1.in‚ 498.80078125" disk.sdb2.out‚1204.45068359375! disk.dm-0.out‚114744.71484375 disk.sda2.in‚ 6.08203125 disk.rbd2.out‚ 2.74609375 disk.md0p1.in‚ 9.26953125# disk.nvme0n1.out‚57080.74609375! proc.memory.rssMB‚ 3001.234375 disk.rbd0.out‚ 2.74609375 disk.nvme1n1.out‚ 57663.96875 disk.md0.out‚ 1160.9765625( proc.memory.availableMB‚ 1290.8671875©ðß~4ž&/ _wandbruntime‚10520  _runtime‚10520ÊâdÊŸ&Ä trainer/global_step‚9049 % _timestamp‚1.7885208946913464e+09 _runtime‚10520.161698911 _step‚741 ' lr-AdamW/pg1‚0.00015494930357492627 ' lr-AdamW/pg2‚0.00015494930357492627£¸tÏ &É ' lr-AdamW/pg1‚0.00015494930357492627 ' lr-AdamW/pg2‚0.00015494930357492627 trainer/global_step‚9049 % _timestamp‚1.7885208946913464e+09 _runtime‚10520.161698911 _step‚741åC¡ª¼4¡&/ _wandbruntime‚10520  _runtime‚10520e¢gŠ ¢&P JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚638 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955591168 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 228859904 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 227053946 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955591168 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 116171895 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚54318953 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚832812488018944 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚754594124532224 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177047040 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚727690021556304 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 186824270 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 252800000 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_^l†ywbatch_start/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 227053946 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 401960960 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10289595325440 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚737941276953882 _step‚742 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚754594124532224 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚754593207121408 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10289595325440 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 _runtime‚10520.163028897 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚744304511203840 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚636 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214665 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10288695917568 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚727690038595664 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411420 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 413878218 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 170490846 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 227027858 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10288695917568 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26088 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7885208946925712e+09 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚737940362503102 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10250340946798 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚744304529206784 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450780 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10251238358218 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚744304529206784 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410816 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10744277547008 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 186824272 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1187 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚832812464078848 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚737940362503102 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10250340946798 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚744304529206784 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450780 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10251238358218 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚744304529206784 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410816 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10744277547008 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 186824272 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1187 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚832812464078848 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚638 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955591168 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 228859904 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 227053946 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955591168 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 116171895 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚54318953 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚832812488018944 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚754594124532224 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177047040 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚727690021556304 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 186824270 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 252800000 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 227053946 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 401960960 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10289595325440 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚737941276953882 _step‚742 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚754594124532224 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚754593207121408 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10289595325440 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 _runtime‚10520.163028897 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚744304511203840 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0æ#P4¤&/ _wandbruntime‚10521  _runtime‚10521Âße·'¥&ŠN OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 227052449 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31686 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚744384159502336 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 186844170 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚738019451258548 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 227052449 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955591168 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚637 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚54324710 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚744384159502336 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10289808720896 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007241 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177047040 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10290708142080 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 116184269 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚635 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 % _timestamp‚1.7885208957499878e+09 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚754674867644416 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚754673950220288 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 413896617 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26114 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10252347019744 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 413922733 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 227078563 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10290708142080 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1187 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚727768001650432 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 186844170 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899421184 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 170509616 _runtime‚10521.22236645 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 413896617 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 186844168 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10745654024192 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 186844168 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10289808720896 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 401960960 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚832901807239680 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10745416788992 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚744384141499392 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 170508979 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 116184271 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411628 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 413922733 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 227078563 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚754674867644416 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899421184 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚744384141499392 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 261175296 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚843647200088576 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10251449608116 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚832901783299584 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450988 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚637 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚54324710 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚744384159502336 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10289808720896 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007241 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177047040 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10290708142080 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 116184269 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚635 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 % _timestamp‚1.7885208957499878e+09 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚754674867644416 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚754673950220288 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 413896617 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26114 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10252347019744 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 413922733 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 227078563 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10290708142080 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1187 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚727768001650432 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 186844170 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899421184 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 170509616 _runtime‚10521.22236645 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 413896617 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 186844168 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10745654024192 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 186844168 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10289808720896 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 401960960 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚832901807239680 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10745416788992 4°v0ÿ.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚744384141499392 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 170508979 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 116184271 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411628 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 413922733 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 227078563 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚754674867644416 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899421184 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚744384141499392 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 261175296 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚843647200088576 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10251449608116 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚832901783299584 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450988 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 PDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 228286848 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚741894813866610 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10306685015326 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 416079930 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410816 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚748286025981440 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚748286043984384 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚848081167094272 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 _runtime‚10576.875344808 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚731589043302064 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚54608101 % _timestamp‚1.7885209514048684e+09 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚837278449055744 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 187819172 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10345249905664 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407872 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 171398796 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214665 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955591168 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26088 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚612 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚848081405214208 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007241 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917410816 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚614 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚741895728317390 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 171398182 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 _step‚746 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10344350497792 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚748286025981440 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 416106020 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 116790695 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚758630376479232 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚748286043984384 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 238119936 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚837278425115648 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 187819170 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 116790693 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚758631293890048 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10802956158464 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1187 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10344350497792 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 187819172lrQ}…!·&†P ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚748286043984384 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 238119936 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚837278425115648 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 187819170 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 116790693 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚758631293890048 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10802956158464 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1187 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10344350497792 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 187819172 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚758630376479232 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 228286848 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚741894813866610 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10306685015326 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 416079930 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410816 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚748286025981440 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚748286043984384 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚848081167094272 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 _runtime‚10576.875344808 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚731589043302064 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚54608101 % _timestamp‚1.7885209514048684e+09 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚837278449055744 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 187819172 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10345249905664 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407872 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 171398796 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214665 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955591168 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26088 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚612 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚848081405214208 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007241 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917410816 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚614 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚741895728317390 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 171398182 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 _step‚746 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10344350497792 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚748286025981440 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 416106020 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 116790695 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚758630376479232êˆÀ_E4¸&/ _wandbruntime‚10577  _runtime‚10577¾8è'¹&‹N YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10306896265224 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚614 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 116803069 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1187 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 242300928 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚54613143 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚747 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 228311465 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 228285351 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚54613755 _runtime‚10577.978111824 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177047040 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚758712037002240 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚741973902622056 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 % _timestamp‚1.7885209525048704e+09 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 416150535 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 416124419 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 228311465 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177047040 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26114 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 116803071 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚758711119578112 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26114 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10345463301120 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31686 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚848171900734464 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 416124419 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚748365656276992 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 401960960 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10804132457984 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚748365656276992 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚748365674279936 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚612 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10307793676852 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 171416212 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚837367744336384 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚758712037002240 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚748365674279936 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10346362722304 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 416150535 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚731667023396192 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 218360832 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚758712037002240 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚748365674279936 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10346362722304 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 416150535 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚731667023396192 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 218360832 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚614 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 116803069 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1187 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 242300928 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚54613143 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚747 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 228311465 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 228285351 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚54613755 _runtime‚10577.978111824 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177047040 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚758712037002240 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚741973902622056 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 % _timestamp‚1.7885209525048704e+09 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 416150535 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 416124419 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 228311465 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177047040 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26114 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 116803071 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚758711119578112 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26114 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10345463301120 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31686 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚848171900734464 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 416124419 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚748365656276992 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 401960960 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10804132457984 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚748365656276992 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚748365674279936 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚612 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10307793676852 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 171416212 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚837367744336384 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 FDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚752267540759040 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 188814072 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410816 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚752267558761984 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1212 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _runtime‚10634.234809674 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26088 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 172306255 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955591168 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 236022784 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚752267540759040 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 188814070 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26090 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 229519750 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10362131672434 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411420 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 229493662 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 % _timestamp‚1.7885210087643945e+09 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955591168 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1187 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚852605684646912 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10361234261014 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 418333822 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚639 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007241 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚762667545837056 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 401960960 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177047040 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚762667545837056 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 229493662 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10400904485888 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955591168 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 236022784 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚752267540759040 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 188814070 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26090 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 229519750 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10362131672434 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411420 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 229493662 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 % _timestamp‚1.7885210087643945e+09 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955591168 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1187 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚852605684646912 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10361234261014 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 418333822 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚639 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007241 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚762667545837056 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 401960960 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177047040 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚762667545837056 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 229493662 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10400904485888 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚752267540759040 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 188814072 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410816 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚752267558761984 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1212 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _runtime‚10634.234809674 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26088 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 172306255 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 JDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 172324371 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚841833729313280 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚735566011063232 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177047040 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 188833970 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 188833968 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚745929268436552 % _timestamp‚1.7885210097056015e+09 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚752347189057536 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 229544367 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10401117881344 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 188833970 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 117421869 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚852695936930304 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 418352221 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10401117881344 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚54902500 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10862231557120 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899421184 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899421184 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10363240333960 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 172323749 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 188833968 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚762749206360064 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚735566028102592 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 trainer/global_step‚9149 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31686 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚54901880 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10862449917952 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 418378337 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411628 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917424128 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚752347171054592 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10402017302528 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 242300928 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26114 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 418378337 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚762748288935936 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 229518253 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚622 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 117421871 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007241 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 229544367 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955591168 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10402017302528 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚762749206360064 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177047040 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26114 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607!ºO1Î&N J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31686 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚54901880 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10862449917952 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 418378337 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411628 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917424128 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚752347171054592 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10402017302528 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 242300928 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26114 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 418378337 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚762748288935936 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 229518253 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚622 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 117421871 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007241 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 229544367 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955591168 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10402017302528 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚762749206360064 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177047040 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚U•Eä607 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26114 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 229518253 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10362342922332 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 _runtime‚10635.17653826 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 172324371 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚841833729313280 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚735566011063232 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177047040 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 188833970 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 188833968 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚745929268436552 % _timestamp‚1.7885210097056015e+09 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚752347189057536 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 229544367 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10401117881344 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 188833970 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 117421869 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚852695936930304 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 418352221 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10401117881344 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚54902500 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10862231557120 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899421184 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899421184 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10363240333960 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 172323749 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 188833968 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚762749206360064 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚735566028102592 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 trainer/global_step‚9149ïb¤q4Ï&/ _wandbruntime‚10635  _runtime‚10635²ÆXî\Ð&Ö 2loss/SRqqt3/assignment_loss‚0.9098718720048947 1loss/SRqqt3/detection_loss‚1.5658409826738655 /loss/FBt3/detection_loss‚2.0158154517412186 epoch‚7 2loss/SRqqt2/assignment_loss‚0.7269833187821011 % _timestamp‚1.7885210097080073e+09 0loss/FRt1/assignment_loss‚2.0726479809239224 1loss/FBt4/assignment_loss‚0.18810685450443998 /loss/FRt3/assignment_loss‚1.723390519618988 2loss/SRqqt4/assignment_loss‚1.7070291060144487 .loss/FRt4/detection_loss‚2.941814913749695 _step‚752 0loss/FBt3/assignment_loss‚0.2679033449145815 1loss/SRqqt2/detection_loss‚0.7471039921542009 .loss/FBt2/detection_loss‚0.535596774762962 0loss/FRt2/assignment_loss‚1.0148612821803373 /loss/FRt4/assignment_loss‚3.762676248550415 1loss/FBt1/assignment_loss‚0.10225143012905846 /loss/FRt3/detection_loss‚2.3137100670486688 1loss/SRqqt1/detection_loss‚1.6100578194155413 trainer/global_step‚9149 2loss/SRqqt1/assignment_loss‚1.3315268462779475 1loss/FBt2/assignment_loss‚0.09958627808950951 .loss/FRt2/detection_loss‚1.632636701797738 %loss/total_loss‚37.22212504700313 _runtime‚10635.178119517 /loss/FRt1/detection_loss‚1.8956854153529659 0loss/SRqqt4/detection_loss‚2.487223428228627 /loss/FBt1/detection_loss‚2.5211112082004545 .loss/FBt4/detection_loss‚3.048693209886551ŠÚá^aÑ&Û _runtime‚10635.178119517 /loss/FRt1/detection_loss‚1.8956854153529659 0loss/SRqqt4/detection_loss‚2.487223428228627 /loss/FBt1/detection_loss‚2.5211112082004545 .loss/FBt4/detection_loss‚3.048693209886551 2loss/SRqqt3/assignment_loss‚0.9098718720048947 1loss/SRqqt3/detection_loss‚1.5658409826738655 /loss/FBt3/detection_loss‚2.0158154517412186 epoch‚7 2loss/SRqqt2/assignment_loss‚0.7269833187821011 % _timestamp‚1.7885210097080073e+09 0loss/FRt1/assignment_loss‚2.0726479809239224 1loss/FBt4/assignment_loss‚0.18810685450443998 /loss/FRt3/assignment_loss‚1.723390519618988 2loss/SRqqt4/assignment_loss‚1.7070291060144487 .loss/FRt4/detection_loss‚2.941814913749695 _step‚752 0loss/FBt3/assignment_loss‚0.2679033449145815 1loss/SRqqt2/detection_loss‚0.7471039921542009 .loss/FBt2/detection_loss‚0.535596774762962 0loss/FRt2/assignment_loss‚1.0148612821803373 /loss/FRt4/assignment_loss‚3.762676248550415 1loss/FBt1/assignment_loss‚0.10225143012905846 /loss/FRt3/detection_loss‚2.3137100670486688 1loss/SRqqt1/detection_loss‚1.6100578194155413 trainer/global_step‚9149 2loss/SRqqt1/assignment_loss‚1.3315268462779475 1loss/FBt2/assignment_loss‚0.09958627808950951 .loss/FRt2/detection_loss‚1.632636701797738 %loss/total_loss‚37.22212504700313ðù;ic”Ò&‚(:‰ ÀÔêÔÞ€ùý gpu.0.gpu‚55.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚39.0 gpu.0.powerWatts‚173.059' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.26475 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Eìd›bÓ&‚(:× ÀÔêԉ뻂 disk.nvme0n1.in‚ 619.5859375 disk.nvme1n1.in‚ 498.80078125 disk.sdb2.out‚ 1219.7734375 disk.sda2.out‚ 1219.7734375# disk.nvme1n1.out‚59611.31640625$ memory_percent‚92.10324287414551 network.sent‚ 2.384456e+07) disk./.usagePercent‚75.84452572071801 disk.rbd2.in‚0 disk.rbd2.out‚ 2.76171875 disk.sda1.out‚0 disk.sda.in‚ 6.08203125 disk.sda.out‚ 1219.7734375 disk.rbd0.in‚0 disk.dm-0.in‚ 1118.38671875 disk.sdb.in‚3.1875 disk.sdb.out‚ 1219.7734375 disk.sda2.in‚ 6.08203125* proc.memory.percent‚18.320035934448242 disk.rbd0.out‚ 2.8203125 disk.md0.out‚ 1175.7265625" proc.memory.rssMB‚ 3001.5546875 network.recv‚ 3.398629e+06 disk.md0p1.out‚ 1175.7265625 disk.sdb1.in‚0 disk.sdb1.out‚0( proc.memory.availableMB‚ 1293.8046875 cpu‚25.206193693003584 disk.sdb2.in‚3.1875 disk.sda1.in‚0 disk.dm-0.out‚ 116692.921875 disk.md0.in‚ 9.26953125 proc.cpu.threads‚17# disk.nvme0n1.out‚57081.60546875$ disk./.usageGB‚2712.567611694336 disk.md0p1.in‚ 9.26953125À‡Óƒ”Ô&‚(:‰ ÏÔêÔ÷ý¶ÿ gpu.0.gpu‚45.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚170.559' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.63975 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0âÇö rÕ&‚(:ç ÏÔêÔçÝìÿ network.sent‚ 2.3929433e+07 disk.nvme1n1.in‚ 498.80078125 disk.sdb1.in‚0 disk.sdb2.in‚3.1875 disk.sda1.in‚0) disk./.usagePercent‚75.85150430601367 disk.sdb.in‚3.1875 cpu‚25.205893947156614 proc.cpu.threads‚17 disk.sda.in‚ 6.08203125! disk.nvme1n1.out‚ 59807.453125 disk.sda1.out‚0* proc.memory.percent‚18.320035934448242! disk.dm-0.out‚116889.06640625 disk.md0p1.out‚ 1177.57421875 disk.sda.out‚1221.6982421875! disk.sda2.out‚1221.6982421875 disk.rbd0.in‚0 disk.md0.out‚ 1177.57421875 disk.rbd2.in‚0 disk.sdb.out‚1221.6982421875 disk.md0p1.in‚ 9.26953125# disk.nvme0n1.out‚57081.61328125% disk./.usageGB‚2712.8171997070312 disk.rbd2.out‚ 2.82421875 disk.dm-0.in‚ 1118.38671875$ memory_percent‚92.10324287414551 network.recv‚ 3.403855e+06 disk.sdb1.out‚0 disk.nvme0n1.in‚ 619.5859375( proc.memory.availableMB‚ 1293.8046875" proc.memory.rssMB‚ 3001.5546875 disk.sda2.in‚ 6.08203125 disk.rbd0.out‚ 2.8203125 disk.md0.in‚ 9.26953125! disk.sdb2.out‚1221.6982421875-UÝŽ“Ö&‚(:ˆ ÞÔêÔ«šÿ gpu.0.gpu‚56.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚116.306' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚29.0765 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0W[âÀw×&‚(:ì ÞÔêÔµªëÿ disk.rbd0.out‚ 2.92236328125 disk.sdb.out‚1223.2607421875% disk./.usageGB‚2712.7265548706055 disk.dm-0.in‚ 1118.38671875 disk.md0p1.out‚ 1179.06640625" proc.memory.rssMB‚ 3001.5546875 disk.sda.in‚ 6.08203125 disk.sdb1.in‚0 network.recv‚ 3.408744e+06 disk.sda.out‚1223.2607421875 disk.md0.out‚ 1179.06640625 disk.sda1.in‚0! disk.dm-0.out‚116890.44140625 disk.sdb.in‚3.1875# memory_percent‚92.1046495437622* proc.memory.percent‚18.320035934448242 disk.nvme0n1.in‚ 619.5859375) disk./.usagePercent‚75.84896983844939! disk.sda2.out‚1223.2607421875" disk.nvme1n1.out‚ 59808.7421875 disk.md0.in‚ 9.26953125 disk.rbd2.in‚0 cpu‚25.205605458239717 disk.nvme1n1.in‚ 498.80078125! disk.sdb2.out‚1223.2607421875 disk.sda1.out‚0# disk.nvme0n1.out‚57081.69921875 proc.cpu.threads‚17 disk.sdb1.out‚0 disk.sdb2.in‚3.1875 disk.md0p1.in‚ 9.26953125) proc.memory.availableMB‚ 1293.57421875 disk.rbd0.in‚0 disk.rbd2.out‚ 2.82421875 network.sent‚ 2.3937488e+07 disk.sda2.in‚ 6.08203125qôþ4Ø&/ _wandbruntime‚10686  _runtime‚10686•¥|„ÊÙ&Ä ' lr-AdamW/pg1‚0.00014091942157993487 ' lr-AdamW/pg2‚0.00014091942157993487 trainer/global_step‚9199 % _timestamp‚1.7885210611241202e+09 _runtime‚10686.594444139 _step‚753üO‚nÏÚ&É % _timestamp‚1.7885210611241202e+09 _runtime‚10686.594444139 _step‚753 ' lr-AdamW/pg1‚0.00014091942157993487 ' lr-AdamW/pg2‚0.00014091942157993487 trainer/global_step‚9199ñ…æ){4Û&/ _wandbruntime‚10686  _runtime‚10686çpÙ (Ü&P [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10455659658240 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 236022784 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10919628576768 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1212 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 % _timestamp‚1.7885210611251118e+09 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚766705632605696 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚606 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 173214015 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214665 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 230752652 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26088 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10455659658240 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 189808972 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚756249073539584 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407872 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 420561624 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177047040 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚749803716593626 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚766704715194880 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10416680918122 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 189808972 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 230726564 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450780 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚739387035675504 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 230726564 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚756249055536640 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚55186328 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚857130211788800 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10417578329542 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚756249055536640 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 173214623 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411420 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 420535534 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955591168 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10456559066112 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728’ð­Ý&†P WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1212 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 % _timestamp‚1.7885210611251118e+09 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚766705632605696 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚606 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 173214015 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214665 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 230752652 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26088 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10455659658240 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 189808972 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚756249073539584 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407872 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 420561624 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177047040 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚749803716593626 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚766704715194880 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10416680918122 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 189808972 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 230726564 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450780 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚739387035675504 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_\êG_pool.peak‚ 12392107008 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 230726564 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚756249055536640 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚55186328 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚857130211788800 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10417578329542 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚756249055536640 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 173214623 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411420 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 420535534 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955591168 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10456559066112 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10455659658240 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 236022784 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10919628576768 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564òÐdÓÇ4Þ&/ _wandbruntime‚10687  _runtime‚10687ù•$ð'ß&‹N K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955591168 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177047040 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚749882805349072 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 % _timestamp‚1.7885210620635366e+09 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 420606139 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚766785458293760 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10456772461568 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚846299666409984 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 420580023 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411628 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007241 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 230751155 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1187 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚857220699786752 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 224652288 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 173232708 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10921009436672 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚628 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26114 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 _runtime‚10687.534541808 trainer/global_step‚9199 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899421184 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 189828868 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚846299690350080 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚55192037 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚749883719800060 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚55191409 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31686 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 118040671 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 118040669 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚766786375717888 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 401960960 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10417789579440 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26114 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10457671882752 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899421184 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚857220451194368 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 189828868 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 173232078 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 420606139 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚756328703835136 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚739465015769632 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 189828870 _step‚755 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚756328685832192 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917424128 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 189828870 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10920784784384 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚756328703835136 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 230751155 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚766786375717888 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚630 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10457671882752 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 420606139 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚766785458293760 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10456772461568 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚846299666409984 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 420580023 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411628 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007241 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 230751155 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1187 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚857220699786752 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 224652288 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 173232708 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10921009436672 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚628 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26114 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 _runtime‚10687.534541808 trainer/global_step‚9199 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899421184 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 189828868 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚846299690350080 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚55192037 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚749883719800060 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚55191409 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31686 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 118040671 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 118040669 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚766786375717888 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 401960960 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10417789579440 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26114 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10457671882752 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899421184 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚857220451194368 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 189828868 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 173232078 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 420606139 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚756328703835136 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚739465015769632 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 189828870ó:5¦4á&/ _wandbruntime‚10687  _runtime‚10687Ê«fÑYâ&Ó /loss/FRt3/detection_loss‚2.0728352073320124 1loss/SRqqt2/detection_loss‚0.8807928501628339 1loss/SRqqt3/detection_loss‚1.6318742329983607 .loss/FBt1/detection_loss‚2.011652879990064 _step‚756 1loss/FBt2/assignment_loss‚0.09254838333500828 /loss/FBt3/detection_loss‚2.5974961072206497 0loss/FBt3/assignment_loss‚0.1217144001991528 %loss/total_loss‚37.67505664698308 /loss/FRt3/assignment_loss‚2.219428293926771 1loss/FBt4/assignment_loss‚0.22536342718875252 _runtime‚10687.536026845 trainer/global_step‚9199 2loss/SRqqt3/assignment_loss‚1.2079600796265446 /loss/FRt4/assignment_loss‚2.936860767858369 1loss/SRqqt4/assignment_loss‚2.145200862928673 /loss/FRt1/detection_loss‚2.0443333649992175 .loss/FRt2/detection_loss‚1.736332390958873 0loss/SRqqt4/detection_loss‚2.458886243678905 .loss/FBt2/detection_loss‚0.559642215937744 2loss/SRqqt1/assignment_loss‚1.2056437605060637 1loss/SRqqt1/detection_loss‚1.6031117006981124 $ _timestamp‚1.788521062065915e+09 0loss/FRt1/assignment_loss‚2.5441740384468665 .loss/FRt4/detection_loss‚2.844833412340709 /loss/FBt4/detection_loss‚2.2959677537282306 epoch‚7 0loss/FRt2/assignment_loss‚1.1497788727283478 2loss/SRqqt2/assignment_loss‚0.7513195367064327 0loss/FBt1/assignment_loss‚0.33730586348638363I'^ã&Ø .loss/FRt2/detection_loss‚1.736332390958873 0loss/SRqqt4/detection_loss‚2.458886243678905 .loss/FBt2/detection_loss‚0.559642215937744 2loss/SRqqt1/assignment_loss‚1.2056437605060637 1loss/SRqqt1/detection_loss‚1.6031117006981124 $ _timestamp‚1.788521062065915e+09 0loss/FRt1/assignment_loss‚2.5441740384468665 .loss/FRt4/detection_loss‚2.844833412340709 /loss/FBt4/detection_loss‚2.2959677537282306 epoch‚7 0loss/FRt2/assignment_loss‚1.1497788727283478 2loss/SRqqt2/assignment_loss‚0.7513195367064327 0loss/FBt1/assignment_loss‚0.3373058634863836 /loss/FRt3/detection_loss‚2.0728352073320124 1loss/SRqqt2/detection_loss‚0.8807928501628339 1loss/SRqqt3/detection_loss‚1.6318742329983607 .loss/FBt1/detection_loss‚2.011652879990064 _step‚756 1loss/FBt2/assignment_loss‚0.09254838333500828 /loss/FBt3/detection_loss‚2.5974961072206497 0loss/FBt3/assignment_loss‚0.1217144001991528 %loss/total_loss‚37.67505664698308 /loss/FRt3/assignment_loss‚2.219428293926771 1loss/FBt4/assignment_loss‚0.22536342718875252 _runtime‚10687.536026845 trainer/global_step‚9199 2loss/SRqqt3/assignment_loss‚1.2079600796265446 /loss/FRt4/assignment_loss‚2.936860767858369 1loss/SRqqt4/assignment_loss‚2.145200862928673 /loss/FRt1/detection_loss‚2.0443333649992175ôHj–”ä&‚(:‰ íÔêÔš€ gpu.0.gpu‚55.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚37.0 gpu.0.powerWatts‚168.383' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.09575 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 %–Dxå&‚(:í íÔêÔ®“€! disk.dm-0.out‚117106.49609375 disk.sda.in‚ 6.08203125 disk.rbd0.out‚ 2.92236328125" disk.nvme1n1.out‚ 60024.6640625 disk.md0.out‚ 1180.671875 disk.dm-0.in‚ 1118.38671875" proc.memory.rssMB‚ 3001.5546875 disk.sdb1.in‚0 disk.md0p1.out‚ 1180.671875 disk.sdb.in‚3.1875 network.recv‚ 3.412061e+06 disk.md0.in‚ 9.26953125" disk.sdb2.out‚1224.93310546875 disk.sda1.in‚0 disk.sda1.out‚0 disk.nvme0n1.in‚ 619.5859375$ memory_percent‚92.10476875305176# disk.nvme0n1.out‚57081.83203125! disk.sda.out‚1224.93310546875" disk.sda2.out‚1224.93310546875 network.sent‚ 2.3940542e+07$ disk./.usageGB‚2712.867374420166 disk.rbd2.in‚0 disk.rbd2.out‚ 2.91162109375 disk.md0p1.in‚ 9.26953125( proc.memory.availableMB‚ 1293.5546875 cpu‚25.20531205761879 proc.cpu.threads‚17 disk.nvme1n1.in‚ 498.80078125 disk.sdb2.in‚3.1875! disk.sdb.out‚1224.93310546875 disk.rbd0.in‚0* proc.memory.percent‚18.320035934448242) disk./.usagePercent‚75.85290721199267 disk.sdb1.out‚0 disk.sda2.in‚ 6.08203125¹jT´”æ&‚(:‰ üÔêÔ“²šÿ gpu.0.gpu‚55.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚174.109' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.52725 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0BBóxç&‚(:í üÔêÔ•½…€ network.recv‚ 3.417503e+06% disk./.usageGB‚2712.8670806884766) proc.memory.availableMB‚ 1293.55078125 cpu‚25.205029901987295 disk.rbd0.in‚0# disk.nvme1n1.out‚60257.12890625 disk.dm-0.in‚ 1118.38671875 disk.sdb.in‚3.1875$ memory_percent‚92.10479259490967 proc.cpu.threads‚17 disk.sdb1.out‚0 disk.md0.in‚ 9.26953125! disk.sdb2.out‚1226.2431640625! disk.dm-0.out‚117339.11328125 disk.md0p1.in‚ 9.26953125 disk.nvme0n1.in‚ 619.5859375" proc.memory.rssMB‚ 3001.5546875 disk.rbd2.in‚0 disk.rbd2.out‚ 2.91162109375 disk.sda1.in‚0( disk./.usagePercent‚75.8528989991317! disk.sda2.out‚1226.2431640625 disk.rbd0.out‚ 2.9619140625 disk.sdb2.in‚3.1875 network.sent‚ 2.4048469e+07 disk.sda.in‚ 6.08203125 disk.sda.out‚1226.2431640625 disk.sda2.in‚ 6.08203125 disk.sda1.out‚0 disk.sdb.out‚1226.2431640625* proc.memory.percent‚18.320035934448242 disk.nvme1n1.in‚ 498.80078125 disk.sdb1.in‚0 disk.md0.out‚ 1181.92578125 disk.md0p1.out‚ 1181.92578125! disk.nvme0n1.out‚ 57081.984375„ÛÍ”è&‚(:‰ ‹ÕêÔ ÷‰€ gpu.0.gpu‚29.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚161.587' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚40.39675 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0»é˜xé&‚(:í ‹ÕêÔù¨Ó network.recv‚ 3.421661e+06 disk.sdb1.in‚0 disk.rbd2.out‚ 2.974609375" disk.sdb2.out‚1228.19677734375! disk.dm-0.out‚117394.83984375 disk.nvme0n1.in‚ 619.5859375# disk.nvme0n1.out‚57081.99609375$ memory_percent‚92.10402965545654 disk.md0p1.out‚ 1183.82421875! disk.sdb.out‚1228.19677734375 disk.nvme1n1.in‚ 498.80078125% disk./.usageGB‚2712.8713912963867 disk.dm-0.in‚ 1118.38671875 disk.md0p1.in‚ 9.26953125 cpu‚25.204735995013603 network.sent‚ 2.4054287e+07! disk.sda.out‚1228.19677734375 disk.rbd0.out‚ 2.9619140625 disk.nvme1n1.out‚ 60312.84375 disk.rbd0.in‚0' disk./.usagePercent‚75.853019525533 disk.md0.out‚ 1183.82421875 disk.sda1.in‚0 disk.sdb.in‚3.1875) proc.memory.availableMB‚ 1293.67578125 disk.rbd2.in‚0 disk.sdb2.in‚3.1875 disk.sda2.in‚ 6.08203125" proc.memory.rssMB‚ 3001.5546875* proc.memory.percent‚18.320035934448242 disk.sdb1.out‚0 disk.md0.in‚ 9.26953125 proc.cpu.threads‚17 disk.sda1.out‚0 disk.sda.in‚ 6.08203125" disk.sda2.out‚1228.19677734375”ʵ³”ê&‚(:‰ šÕêÔ‡²—ÿ gpu.0.gpu‚53.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚170.219' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.55475 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚02¢&øpë&‚(:å šÕêÔ¦ÞÛÿ disk.sda.in‚ 6.08203125 disk.md0.in‚ 9.26953125 disk.sda1.in‚0 disk.md0p1.in‚ 9.26953125 disk.md0p1.out‚ 1185.3359375( proc.memory.availableMB‚ 1293.3203125* proc.memory.percent‚18.320035934448242 cpu‚25.20443692603235 network.sent‚ 2.4062795e+07 disk.sdb2.out‚1229.751953125 disk.sda1.out‚0 disk.sdb.out‚1229.751953125" disk.nvme0n1.out‚ 57082.1328125% disk./.usageGB‚2712.5917892456055 disk.sda.out‚1229.751953125 disk.rbd0.out‚ 3.02490234375 disk.nvme1n1.in‚ 498.80078125" proc.memory.rssMB‚ 3001.5546875 proc.cpu.threads‚17 network.recv‚ 3.426313e+06( disk./.usagePercent‚75.8452017351706# disk.nvme1n1.out‚60482.84765625 disk.rbd2.in‚0 disk.sda2.in‚ 6.08203125 disk.rbd0.in‚0 disk.sdb1.in‚0 disk.dm-0.in‚ 1118.38671875 disk.sdb.in‚3.1875$ memory_percent‚92.10619926452637 disk.sdb1.out‚0 disk.sdb2.in‚3.1875 disk.nvme0n1.in‚ 619.5859375 disk.sda2.out‚1229.751953125 disk.md0.out‚ 1185.3359375 disk.rbd2.out‚ 2.974609375! disk.dm-0.out‚117564.98046875÷=±p4ì&/ _wandbruntime‚10743  _runtime‚10743ÉvŽHÊí&Ä _step‚757 ' lr-AdamW/pg1‚0.00013636151571984545 ' lr-AdamW/pg2‚0.00013636151571984545 trainer/global_step‚9249 % _timestamp‚1.7885211176706924e+09 _runtime‚10743.141055112y¸×UÏî&É ' lr-AdamW/pg1‚0.00013636151571984545 ' lr-AdamW/pg2‚0.00013636151571984545 trainer/global_step‚9249 % _timestamp‚1.7885211176706924e+09 _runtime‚10743.141055112 _step‚757õ/ƒõ4ï&/ _wandbruntime‚10743  _runtime‚10743á%žš¦ð&P YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚760230570314240 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10512213646336 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚850676332166144 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 \a$HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚760230570314240 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚861654534703616 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚753759082407914 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10473024986650 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26088 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26088 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚55475141 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10978226477568 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚770741884552704 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10512213646336 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 218374144 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚758 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 190803872 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1187 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚850676308226048 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 422763336 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚760230588317184 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214665 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 422789426 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚620 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚861654777017856 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26090 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚770742801963520 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 118647095 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _runtime‚10743.142387078 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 190803870 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 231985554 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955591168 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1212 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407872 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410816 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007241 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 231985554 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 401960960 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚753758167957134 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411420 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 231985554 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 401960960 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚753758167957134 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411420 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚850676332166144 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚760230570314240 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚861654534703616 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚753759082407914 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10473024986650 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26088 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26088 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚55475141 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10978226477568 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚770741884552704 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10512213646336 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 218374144 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚758 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 190803872 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1187 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚850676308226048 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 422763336 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚760230588317184 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214665 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 422789426 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚620 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚861654777017856 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26090 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚770742801963520 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 118647095 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _runtime‚10743.142387078 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 190803870 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 231985554 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955591168 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1212 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407872 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410816 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35öƒ‘¤4ò&/ _wandbruntime‚10744  _runtime‚10744è¾+'ó&ˆN ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚760310200609792 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 trainer/global_step‚9249 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10513326462976 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 190823770 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177047040 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚861745030302720 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411628 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 422833941 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 174139730 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚753837256712580 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 422807825 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 118659469 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 232010171 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 231984057 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 190823770 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26114 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1187 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10979619119616 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚595 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚760310218612736 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31686 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚743364020476032 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 232010171 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955591168 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10512427041792 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 401960960 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 " _timestamp‚1.7885211186229e+09 _runtime‚10744.093895152 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 174140327 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚753838171163568 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚743364037515392 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007241 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 422833941 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚770823545075712 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚850765627446784 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 190823768 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 231984057 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450988 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚760310218612736 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899421184 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚861745270506496 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10979402855936 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10473236236548 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899421184 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 190823768 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450988 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚760310218612736 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899421184 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚861745270506496 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10979402855936 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10473236236548 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899421184 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 190823768 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10513326462976 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 190823770 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177047040 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚861745030302720 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411628 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 422833941 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 174139730 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚753837256712580 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 422807825 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 118659469 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 232010171 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 231984057 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 190823770 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26114 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1187 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10979619119616 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚595 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚760310218612736 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31686 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚743364020476032 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 232010171 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955591168 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10512427041792 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 401960960 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 " _timestamp‚1.7885211186229e+09 _runtime‚10744.093895152 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 174140327 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚753838171163568 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚743364037515392 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007241 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 422833941 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚770823545075712 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚850765627446784 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 190823768 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 231984057 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 FDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚764212103094784 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1187 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚866179397899776 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 191798770 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10566968818688 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 252800000 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10566968818688 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚774779053910528 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955591168 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 191798772 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 233192368 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚855142269262848 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚757713533771422 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214665 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007241 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26088 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 233192368 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚764212085091840 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 175029416 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 228859904 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 trainer/global_step‚9299 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407872 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10567868226560 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚866179145099776 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10528471643758 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 _runtime‚10801.852058417 _step‚762 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1212 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 233218456 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚764212103094784 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚774779971321344 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚774779053910528 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚55763523 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚757712619320642 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10527574232338 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 425017228 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 191798772 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 119265895 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 401960960 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚747185062127664 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177047040 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ¢Z y…'†P [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10566968818688 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 252800000 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10566968818688 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚774779053910528 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955591168 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 191798772 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 233192368 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚855142269262848 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚757713533771422 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214665 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007241 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26088 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 233192368 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚764212085091840 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 175029416 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 228859904 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 trainer/global_step‚9299 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407872 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10567868226560 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚866179145099776 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10528471643758 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 _runtime‚10801.852058417 _step‚762 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1212 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 233218456 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚764212103094784 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚774779971321344 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚774779053910528 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚55763523 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚757712619320642 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10527574232338 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 425017228 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 191798772 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 119265895 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 401960960 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚747185062127664 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177047040 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26088 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚55764150 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚764212103094784 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1187 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚866179397899776 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 191798770 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607úˆge4†'/ _wandbruntime‚10803  _runtime‚10803¶…R6'‡'‹N PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 175048090 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 119278269 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 425061743 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 % _timestamp‚1.7885211780032098e+09 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚774859797009408 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11038056907264 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10529580305284 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007241 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚55769819 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 238106624 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚855231588483584 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 191818670 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177047040 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 191818668 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚757791708076088 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚747263042221792 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚764291715387392 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26114 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚774860714433536 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚764291733390336 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899421184 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚866269645390848 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 191818670 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 233216959 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917424128 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899421184 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31686 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚55769201 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚757792622527076 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411628 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10568081622016 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10528682893656 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 425035627 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177047040 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚774860714433536 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚764291733390336 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 425061743 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11038271073792 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 175047470 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚620 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚866269883497472 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 214166528 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450988 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 401960960 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607)eÛ'ˆ'N H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 425061743 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 % _timestamp‚1.7885211780032098e+09 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚774859797009408 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11038056907264 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10529580305284 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007241 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚55769819 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 238106624 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚855231588483584 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 191818670 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177047040 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 191818668 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚757791708076088 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚747263042221792 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚764291715387392 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26114 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚774860714433536 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚764291733390336 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899421184 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚866269645390848 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 191818670 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 233216959 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917424128 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899421184 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31686 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚55769201 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚757792622527076 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411628 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10568081622016 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10528682893656 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 425035627 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177047040 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚774860714433536 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚764291733390336 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 425061743 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11038271073792 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 175047470 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚620 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚866269883497472 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 214166528 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450988 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 401960960 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 175048090 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 119278269 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0ûðÛ™4‰'/ _wandbruntime‚10803  _runtime‚10803{O ÂYŠ'Ó 1loss/FBt2/assignment_loss‚0.05670823143580639 1loss/SRqqt4/detection_loss‚2.6935441446304322 /loss/FBt1/detection_loss‚2.1157007813453674 /loss/FRt1/assignment_loss‚2.520582679372567 2loss/SRqqt2/assignment_loss‚0.8329868640073321 1loss/FBt1/assignment_loss‚0.20857549922366161 1loss/FBt3/assignment_loss‚0.35341254341720213 .loss/FBt3/detection_loss‚2.095933213829994 1loss/FBt4/assignment_loss‚0.41150573209205715 trainer/global_step‚9299 0loss/FRt3/assignment_loss‚2.7614054908143713 2loss/SRqqt1/assignment_loss‚1.4808442469905405 ,loss/FRt1/detection_loss‚1.9215462952232 .loss/FRt4/detection_loss‚2.992272280413529 epoch‚7 % _timestamp‚1.7885211780061138e+09 _step‚764 /loss/FRt4/assignment_loss‚2.907470863440941 2loss/SRqqt4/assignment_loss‚1.3243417727947235 1loss/SRqqt2/detection_loss‚0.8124947227634393 /loss/FBt4/detection_loss‚2.2042407301756053 2loss/SRqqt3/assignment_loss‚1.5157011623743402 /loss/FRt2/detection_loss‚1.5385818970759237 ,loss/FRt3/detection_loss‚2.0979934231715 0loss/SRqqt1/detection_loss‚1.514544601680017 1loss/SRqqt3/detection_loss‚1.9488797925066117 /loss/FBt2/detection_loss‚0.4467946957358543 %loss/total_loss‚37.96871556446471 _runtime‚10803.476523612 0loss/FRt2/assignment_loss‚1.2126538999496945Ÿ]‡^‹'Ø _step‚764 /loss/FRt4/assignment_loss‚2.907470863440941 2loss/SRqqt4/assignment_loss‚1.3243417727947235 1loss/SRqqt2/detection_loss‚0.8124947227634393 /loss/FBt4/detection_loss‚2.2042407301756053 2loss/SRqqt3/assignment_loss‚1.5157011623743402 /loss/FRt2/detection_loss‚1.5385818970759237 ,loss/FRt3/detection_loss‚2.0979934231715 0loss/SRqqt1/detection_loss‚1.514544601680017 1loss/SRqqt3/detection_loss‚1.9488797925066117 /loss/FBt2/detection_loss‚0.4467946957358543 %loss/total_loss‚37.96871556446471 _runtime‚10803.476523612 0loss/FRt2/assignment_loss‚1.2126538999496945 1loss/FBt2/assignment_loss‚0.05670823143580639 1loss/SRqqt4/detection_loss‚2.6935441446304322 /loss/FBt1/detection_loss‚2.1157007813453674 /loss/FRt1/assignment_loss‚2.520582679372567 2loss/SRqqt2/assignment_loss‚0.8329868640073321 1loss/FBt1/assignment_loss‚0.20857549922366161 1loss/FBt3/assignment_loss‚0.35341254341720213 .loss/FBt3/detection_loss‚2.095933213829994 1loss/FBt4/assignment_loss‚0.41150573209205715 trainer/global_step‚9299 0loss/FRt3/assignment_loss‚2.7614054908143713 2loss/SRqqt1/assignment_loss‚1.4808442469905405 ,loss/FRt1/detection_loss‚1.9215462952232 .loss/FRt4/detection_loss‚2.992272280413529 epoch‚7 % _timestamp‚1.7885211780061138e+09ü¥€¹æžŒ'‚(:“ åÕêÔ®ŒÃ€ gpu.0.gpu‚20.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚36.0 gpu.0.powerWatts‚119.368' gpu.0.enforcedPowerLimitWatts‚400.0) gpu.0.powerPercent‚29.841999999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Û‡w=i'‚(:Þ åÕêÔÃŒ® network.sent‚ 2.4193919e+07 disk.sda2.out‚ 1237.9921875 disk.md0.in‚ 9.26953125 disk.sda1.in‚0" disk.nvme0n1.out‚ 57249.2890625 disk.rbd0.out‚ 3.087890625 disk.nvme1n1.in‚ 498.80078125 disk.rbd2.in‚0) proc.memory.availableMB‚ 1293.54296875 network.recv‚ 3.453337e+06) disk./.usagePercent‚75.87050076010561 disk.md0p1.out‚ 1193.28515625! disk.nvme1n1.out‚ 61166.015625 disk.sdb2.out‚ 1237.9921875 disk.sda1.out‚0% disk./.usageGB‚2713.4966049194336 disk.sda.out‚ 1237.9921875 disk.rbd0.in‚0 disk.sdb1.out‚0 disk.sda2.in‚ 6.08203125 disk.sdb1.in‚0 disk.md0.out‚ 1193.28515625 disk.rbd2.out‚ 3.03759765625" proc.memory.rssMB‚ 3001.5546875 disk.sdb.in‚3.1875 disk.sdb.out‚ 1237.9921875 disk.sda.in‚ 6.08203125 disk.sdb2.in‚3.1875 disk.dm-0.out‚118415.3046875$ memory_percent‚92.10484027862549 disk.dm-0.in‚ 1118.38671875* proc.memory.percent‚18.320035934448242 cpu‚25.20315289708635 proc.cpu.threads‚17 disk.md0p1.in‚ 9.26953125 disk.nvme0n1.in‚ 619.5859375,af»’Ž'‚(:‡ ôÕêÔ¡–Æÿ gpu.0.gpu‚37.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚155.684' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚38.921 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0KzÎêp'‚(:å ôÕêÔй€% disk./.usageGB‚2713.6221199035645 disk.sda2.out‚ 1240.31640625 disk.sdb2.out‚ 1240.31640625# disk.nvme0n1.out‚57249.47265625 disk.sdb.in‚3.1875 disk.sda.out‚ 1240.31640625 network.sent‚ 2.4301544e+07" proc.memory.rssMB‚ 3001.5546875 proc.cpu.threads‚17 disk.nvme1n1.in‚ 498.80078125 disk.sdb2.in‚3.1875 cpu‚25.20292270803615 disk.sda2.in‚ 6.08203125 disk.rbd0.in‚0 disk.sda1.out‚0 disk.md0p1.in‚ 9.26953125 disk.rbd0.out‚ 3.087890625# disk.nvme1n1.out‚61166.34765625) proc.memory.availableMB‚ 1293.40234375 network.recv‚ 3.459164e+06 disk.sdb1.in‚0 disk.rbd2.in‚0 disk.dm-0.out‚118415.8203125 disk.nvme0n1.in‚ 619.5859375 disk.sda.in‚ 6.08203125 disk.sdb1.out‚0 disk.md0.in‚ 9.26953125 disk.md0p1.out‚ 1195.54296875 disk.sdb.out‚ 1240.31640625 disk.md0.out‚ 1195.54296875 disk.rbd2.out‚ 3.03759765625 disk.sda1.in‚0 disk.dm-0.in‚ 1118.38671875$ memory_percent‚92.10569858551025* proc.memory.percent‚18.320035934448242) disk./.usagePercent‚75.87401021159401l`냓'‚(:ˆ ƒÖêÔÝ¿´ÿ gpu.0.gpu‚41.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚174.286' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.5715 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ž¢q‘'‚(:æ ƒÖêÔ­‚Œ€! disk.sdb.out‚1241.79833984375* proc.memory.percent‚18.320035934448242 disk.nvme1n1.in‚ 498.80078125 disk.md0.in‚ 9.26953125 disk.sda1.out‚0 disk.dm-0.in‚ 1118.38671875 disk.md0p1.in‚ 9.26953125! disk.sda.out‚1241.79833984375" disk.sda2.out‚1241.79833984375 cpu‚25.20269035648879 disk.rbd0.in‚0 disk.sdb2.in‚3.1875" proc.memory.rssMB‚ 3001.5546875 proc.cpu.threads‚17 disk.rbd2.out‚ 3.03759765625$ memory_percent‚92.10777282714844 disk.rbd0.out‚ 3.087890625 network.recv‚ 3.463298e+06 disk.sdb1.in‚0" disk.nvme0n1.out‚ 57364.9296875 disk.sda.in‚ 6.08203125 disk.rbd2.in‚0) disk./.usagePercent‚75.88096970465442$ disk./.usageGB‚2713.871025085449# disk.nvme1n1.out‚61480.94140625 disk.sdb1.out‚0 disk.md0.out‚ 1196.96875" disk.sdb2.out‚1241.79833984375 disk.sda1.in‚0! disk.dm-0.out‚118833.49609375 disk.md0p1.out‚ 1196.96875 disk.nvme0n1.in‚ 619.5859375 disk.sdb.in‚3.1875 disk.sda2.in‚ 6.08203125% proc.memory.availableMB‚ 1293.0625 network.sent‚ 2.4309773e+07H–n’’'‚(:‡ ’ÖêÔ°ÿ‘ÿ gpu.0.gpu‚13.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚36.0 gpu.0.powerWatts‚114.408' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.602 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚06‰D"“'‚(:â ’ÖêÔÆ âÿ disk.dm-0.in‚ 1118.4375$ disk./.usageGB‚2713.871025085449 disk.rbd0.out‚ 3.087890625 disk.rbd2.out‚ 3.03759765625! disk.sdb2.out‚1243.3798828125 disk.sda1.in‚0* proc.memory.percent‚18.320035934448242 disk.dm-0.out‚ 118982.375" proc.memory.rssMB‚ 3001.5546875) disk./.usagePercent‚75.88096970465442 disk.nvme1n1.in‚ 498.80078125 disk.sdb1.in‚0 disk.md0.in‚ 9.26953125) proc.memory.availableMB‚ 1293.41015625 proc.cpu.threads‚17 network.sent‚ 2.4316056e+¨ßìÌN07 disk.sdb.out‚1243.3798828125 disk.sda2.in‚ 6.08203125 disk.sdb1.out‚0 disk.md0.out‚ 1198.484375 disk.nvme0n1.in‚ 619.63671875 disk.rbd2.in‚0 disk.sdb2.in‚3.1875 disk.md0p1.in‚ 9.26953125 disk.sdb.in‚3.1875! disk.sda2.out‚1243.3798828125 disk.sda1.out‚0$ memory_percent‚92.10565090179443# disk.nvme0n1.out‚57501.18359375 disk.sda.in‚ 6.08203125 disk.sda.out‚1243.3798828125 disk.md0p1.out‚ 1198.484375 cpu‚25.202435606449985 network.recv‚ 3.467814e+06 disk.rbd0.in‚0# disk.nvme1n1.out‚61481.19140625Üõþ4”'/ _wandbruntime‚10861  _runtime‚10861àÁ=ŠÊ•'Ä ' lr-AdamW/pg1‚0.00012742950383052838 ' lr-AdamW/pg2‚0.00012742950383052838 trainer/global_step‚9349 % _timestamp‚1.7885212356196473e+09 _runtime‚10861.090629046 _step‚765²4¹;Ï–'É trainer/global_step‚9349 % _timestamp‚1.7885212356196473e+09 _runtime‚10861.090629046 _step‚765 ' lr-AdamW/pg1‚0.00012742950383052838 ' lr-AdamW/pg2‚0.00012742950383052838ý(çÌ”4—'/ _wandbruntime‚10861  _runtime‚10861YSb(˜'€P ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚768193617872384 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 234451358 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚599 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚761667985134930 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚778817140679168 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26088 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 175937061 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 119884693 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚778816223268352 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407872 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 192793672 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 192793670 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚778817140679168 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1212 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10622623398912 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 236022784 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚761667070684150 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0ˆ¼Û (™'…P ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚870703766675456 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚778817140679168 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26088 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 175937061 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 119884693 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚778816223268352 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407872 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 192793672 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 192793670 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚778817140679168 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1212 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10622623398912 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 236022784 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚761667070684150 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚768193617872384 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 234451358 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚599 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚761667985134930 JDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26114 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 212069376 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 427289545 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1187 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 234475975 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚778897883791360 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 % _timestamp‚1.7885212366494308e+09 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚751162046928192 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 234475975 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899421184 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917424128 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 _runtime‚10862.120301646 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚859697573460480 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚56057490 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _step‚767 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 427289545 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 234449861 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 175954559Oýל'N REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚56057490 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _step‚767 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 427289545 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 234449861 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 175954559 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11096476955648 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚768273248167936 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚870794262485504 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 192813568 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26114 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26114 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 212069376 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 427289545 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1187 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 234475975 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚778897883791360 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 % _timestamp‚1.7885212366494308e+09 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚751162046928192 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 234475975 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899421184 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917424128 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 _runtime‚10862.120301646 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚859697573460480 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728ÿËV•Þ4'/ _wandbruntime‚10862  _runtime‚10862njôÓ]ž'× 0loss/FBt1/assignment_loss‚0.3903584159644587 /loss/FRt4/detection_loss‚3.2045378386974335 1loss/SRqqt1/detection_loss‚1.7796986179053784 0loss/SRqqt3/detection_loss‚1.564276043958259 epoch‚7 1loss/SRqqt4/detection_loss‚2.2489029817721424 _step‚768 /loss/FRt1/detection_loss‚1.9807831377130227 trainer/global_step‚9349 2loss/SRqqt4/assignment_loss‚1.5298147477647837 1loss/SRqqt2/detection_loss‚0.8064709947827984 %loss/total_loss‚39.72690764089751 0loss/FRt2/assignment_loss‚1.7483258818058258 0loss/FBt3/assignment_loss‚0.4006080366534661 /loss/FRt2/detection_loss‚1.7744301488583392 /loss/FRt3/detection_loss‚2.3271287660612616 /loss/FBt2/detection_loss‚0.6562470867639102 .loss/FBt4/detection_loss‚1.769901657104492 0loss/FRt4/assignment_loss‚3.3441449349576775 2loss/SRqqt3/assignment_loss‚1.2328269244286996 1loss/FBt2/assignment_loss‚0.06578595096574076 .loss/FBt1/detection_loss‚2.401029144014631 _runtime‚10862.121300951 % _timestamp‚1.7885212366511438e+09 2loss/SRqqt1/assignment_loss‚1.1764561512693763 2loss/SRqqt2/assignment_loss‚1.0911739380920635 0loss/FBt4/assignment_loss‚0.2904965828754939 .loss/FBt3/detection_loss‚2.961365759372711 0loss/FRt1/assignment_loss‚2.4066887214535573 0loss/FRt3/assignment_loss‚2.5754551776619845(u«bŸ'Ü 2loss/SRqqt4/assignment_loss‚1.5298147477647837 1loss/SRqqt2/detection_loss‚0.8064709947827984 %loss/total_loss‚39.72690764089751 0loss/FRt2/assignment_loss‚1.7483258818058258 0loss/FBt3/assignment_loss‚0.4006080366534661 /loss/FRt2/detection_loss‚1.7744301488583392 /loss/FRt3/detection_loss‚2.3271287660612616 /loss/FBt2/detection_loss‚0.6562470867639102 .loss/FBt4/detection_loss‚1.769901657104492 0loss/FRt4/assignment_loss‚3.3441449349576775 2loss/SRqqt3/assignment_loss‚1.2328269244286996 1loss/FBt2/assignment_loss‚0.06578595096574076 .loss/FBt1/detection_loss‚2.401029144014631 _runtime‚10862.121300951 % _timestamp‚1.7885212366511438e+09 2loss/SRqqt1/assignment_loss‚1.1764561512693763 2loss/SRqqt2/assignment_loss‚1.0911739380920635 0loss/FBt4/assignment_loss‚0.2904965828754939 .loss/FBt3/detection_loss‚2.961365759372711 0loss/FRt1/assignment_loss‚2.4066887214535573 0loss/FRt3/assignment_loss‚2.5754551776619845 0loss/FBt1/assignment_loss‚0.3903584159644587 /loss/FRt4/detection_loss‚3.2045378386974335 1loss/SRqqt1/detection_loss‚1.7796986179053784 0loss/SRqqt3/detection_loss‚1.564276043958259 epoch‚7 1loss/SRqqt4/detection_loss‚2.2489029817721424 _step‚768 /loss/FRt1/detection_loss‚1.9807831377130227 trainer/global_step‚9349€ý&É’ '‚(:‡ ¡ÖêÔúï€ gpu.0.gpu‚47.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚151.95' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚37.9875 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¹3j¡'‚(:ß ¡ÖêÔÄñ² disk.md0.out‚ 1200.4453125 disk.dm-0.in‚ 1118.4375 disk.sda2.out‚1245.396484375 disk.sdb1.in‚0 disk.md0.in‚ 9.26953125 disk.rbd2.out‚ 3.03759765625 disk.sdb2.in‚3.1875 disk.nvme0n1.in‚ 619.63671875* proc.memory.percent‚18.320035934448242# disk./.usageGB‚2713.87113571167 network.recv‚ 3.471307e+06 disk.nvme1n1.in‚ 498.80078125 disk.sdb1.out‚0 disk.sdb2.out‚1245.396484375 proc.cpu.threads‚17# disk.nvme0n1.out‚57633.25390625 disk.sda.in‚ 6.08203125 disk.sda.out‚1245.396484375 disk.rbd0.in‚0 disk.rbd2.in‚0 disk.dm-0.out‚119114.5859375& proc.memory.availableMB‚ 1293.84375" proc.memory.rssMB‚ 3001.5546875 network.sent‚ 2.4331954e+07 disk.rbd0.out‚ 3.087890625 disk.sda1.in‚0 disk.sda1.out‚0 disk.md0p1.in‚ 9.26953125 disk.sdb.in‚3.1875) disk./.usagePercent‚75.88097279780985# disk.nvme1n1.out‚61481.33203125# memory_percent‚92.1030044555664 cpu‚25.202195031309238 disk.sdb.out‚1245.396484375 disk.md0p1.out‚ 1200.4453125 disk.sda2.in‚ 6.08203125Lp ’¢'‚(:‡ °ÖêÔë…ºÿ gpu.0.gpu‚39.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚173.344' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.336 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0hMvw£'‚(:ì °ÖêÔÏ¡œ€ disk.sdb2.in‚3.1875 disk.rbd2.out‚ 3.03759765625" disk.sda2.out‚1246.94384765625 disk.nvme0n1.in‚ 619.63671875 disk.sda1.out‚0 proc.cpu.threads‚17 network.recv‚ 3.477769e+06 disk.rbd0.out‚ 3.087890625 disk.sdb1.in‚0) proc.memory.availableMB‚ 1293.85546875 disk.md0.out‚ 1201.94921875 disk.rbd2.in‚0 disk.sda1.in‚0 disk.nvme0n1.out‚ 57717.03125! disk.sdb.out‚1246.94384765625 disk.rbd0.in‚0 disk.md0.in‚ 9.26953125" disk.sdb2.out‚1246.94384765625 disk.dm-0.in‚ 1118.4375! disk.dm-0.out‚119513.05078125" proc.memory.rssMB‚ 3001.5546875 network.sent‚ 2.4427637e+07) disk./.usagePercent‚75.85151283885624 disk.sdb.in‚3.1875 disk.sda.in‚ 6.08203125 disk.sda2.in‚ 6.08203125 disk.md0p1.in‚ 9.26953125* proc.memory.percent‚18.320035934448242 cpu‚25.20194575291521# disk.nvme1n1.out‚61796.01953125 disk.sdb1.out‚0 disk.md0p1.out‚ 1201.94921875$ memory_percent‚92.10293292999268% disk./.usageGB‚2712.8175048828125! disk.sda.out‚1246.94384765625 disk.nvme1n1.in‚ 498.80078125 2f”¤'‚(:‰ ¿ÖêÔ½¢¦ÿ gpu.0.gpu‚14.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚39.0 gpu.0.powerWatts‚176.461' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚44.11525 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Lœr¥'‚(:ç ¿ÖêÔ‹áØÿ( disk./.usagePercent‚75.8514990796476! disk.sda2.out‚1248.4482421875 disk.rbd0.in‚0 disk.sdb.in‚3.1875* proc.memory.percent‚18.320035934448242 disk.md0.in‚ 9.26953125 disk.md0p1.out‚ 1203.39453125" proc.memory.rssMB‚ 3001.5546875 network.sent‚ 2.4436916e+07 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.sdb2.in‚3.1875! disk.sdb2.out‚1248.4482421875 network.recv‚ 3.488472e+06 disk.sda.in‚ 6.08203125 disk.rbd0.out‚ 3.087890625 disk.md0.out‚ 1203.39453125 disk.rbd2.out‚ 3.03759765625 disk.dm-0.in‚ 1118.4375 disk.md0p1.in‚ 9.26953125( proc.memory.availableMB‚ 1293.8515625# disk.nvme0n1.out‚57845.06640625 disk.sdb.out‚1248.4482421875$ memory_percent‚92.10295677185059 cpu‚25.20171689913904 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-0.out‚ 119729.421875 disk.nvme0n1.in‚ 619.63671875 disk.sda.out‚1248.4482421875 disk.sda2.in‚ 6.08203125# disk.nvme1n1.out‚61884.35546875 disk.rbd2.in‚0% disk./.usageGB‚2712.8170127868652 disk.nvme1n1.in‚ 498.80078125 proc.cpu.threads‚17pþSÅ4¦'/ _wandbruntime‚10915  _runtime‚10915¤¡Ê§'Ä % _timestamp‚1.7885212895605297e+09 _runtime‚10915.030903927 _step‚769 ' lr-AdamW/pg1‚0.00012305725921919064 ' lr-AdamW/pg2‚0.00012305725921919064 trainer/global_step‚9399r/ÌGϨ'É ' lr-AdamW/pg1‚0.00012305725921919064 ' lr-AdamW/pg2‚0.00012305725921919064 trainer/global_step‚9399 % _timestamp‚1.7885212895605297e+09 _runtime‚10915.030903927 _step‚769—­:94©'/ _wandbruntime‚10915  _runtime‚10915!ë (ª'P XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407872 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚618 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚754983071540464 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955591168 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10679177387008 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26088 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10639364957974 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚782853392626176 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 429446742 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10679177387008 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177047040 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚620 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10638467546554 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26088 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 193788570 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 235658172 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 120503495 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚765622436498438 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚754983054501104 _step‚770 % _timestamp‚1.7885212895616179e+09 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 176844712 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚782853392626176 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410816 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411420 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚772175132649984 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 193788572 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚56340599 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955591168 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚772175114647040 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 176844092 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 226762752 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚864074215276544 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11153903382016 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚765621522047658 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚782854310036992 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 trainer/global_step‚9399 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917410816 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 401960960 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 193788570 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10678277979136 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 176844712 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚782853392626176 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410816 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411420 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚772175132649984 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 193788572 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚56340599 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955591168 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚772175114647040 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 176844092 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 226762752 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚864074215276544 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11153903382016 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚765621522047658 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚782854310036992 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 trainer/global_step‚9399 N@DeviceStatsMonitor.on_trainš°‰Gƒ_batch_start/active_bytes.all.current‚ 917410816 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 401960960 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 193788570 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10678277979136 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 JDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚782934135725056 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10679390782464 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177047040 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007241 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 trainer/global_step‚9399 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 176862085 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411628 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 429491231 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 429491231 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 212069376 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450988 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10679390782464 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 120515871 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚875318369016832 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11154858459648 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 235682763 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚782935053149184 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10680290203648 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚56346831 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 401960960 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚755061034595232 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚772254744942592 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚615 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 429517347 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚864163510557184 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚772254744942592 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚772254762945536 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26114 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚56346216 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917424128 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 429517347 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 235708877 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26114 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚617 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11155070529024 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 235682763 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚765700610803104 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10640473619500 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 236009472 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 193808470 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚875318605026304 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 120515869 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955591168 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _step‚771 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177047040™±Z'®'N H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 429517347 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚864163510557184 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚772254744942592 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚772254762945536 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26114 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚56346216 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917424128 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 429517347 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 235708877 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26114 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚617 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11155070529024 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 235682763 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚765700610803104 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10640473619500 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 236009472 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 193808470 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚875318605026304 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 120515869 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955591168 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _step‚771 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177047040 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1187 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10639576207872 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚782934135725056 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10679390782464 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177047040 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007241 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 trainer/global_step‚9399 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 176862085 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411628 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 429491231 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 429491231 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 212069376 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450988 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10679390782464 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 120515871 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚875318369016832 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11154858459648 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 235682763 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚782935053149184 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10680290203648 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚56346831 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 401960960 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚755061034595232 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚772254744942592 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚615 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0ƒg]8 4¯'/ _wandbruntime‚10916  _runtime‚10916Ž±¨]°'× 2loss/SRqqt1/assignment_loss‚1.2724834349385257 1loss/FBt3/assignment_loss‚0.41187456910879516 .loss/FBt4/detection_loss‚2.873715857664744 trainer/global_step‚9399 2loss/SRqqt2/assignment_loss‚1.1527774334747598 % _timestamp‚1.7885212905304549e+09 2loss/SRqqt4/assignment_loss‚1.6463097362291246 1loss/FBt2/assignment_loss‚0.07473987093362928 0loss/SRqqt1/detection_loss‚1.563093294657893 0loss/SRqqt2/detection_loss‚1.012191798142863 /loss/SRqqt4/detection_loss‚2.86048223007293 epoch‚7 /loss/FRt2/detection_loss‚1.7338341017994954 /loss/FRt3/assignment_loss‚2.332912149952679 2loss/SRqqt3/assignment_loss‚0.8101198606465648 1loss/FBt4/assignment_loss‚0.07447656254953472 1loss/SRqqt3/detection_loss‚2.0675710583241975 0loss/FRt4/assignment_loss‚3.4323334978974382 /loss/FRt1/detection_loss‚2.2733126941683808 /loss/FBt1/detection_loss‚2.4157486028141446 /loss/FBt2/detection_loss‚0.5570299926238429 _runtime‚10916.00056056 0loss/FRt1/assignment_loss‚2.0253814065710025 _step‚772 1loss/FBt1/assignment_loss‚0.18178917948130094 /loss/FRt3/detection_loss‚2.4981227028297215 /loss/FRt4/detection_loss‚3.1686470923216445 %loss/total_loss‚39.85553166430546 /loss/FBt3/detection_loss‚2.1849033653736116 0loss/FRt2/assignment_loss‚1.2316811717286402–¯3b±'Ü 1loss/FBt2/assignment_loss‚0.07473987093362928 0loss/SRqqt1/detection_loss‚1.563093294657893 0loss/SRqqt2/detection_loss‚1.012191798142863 /loss/SRqqt4/detection_loss‚2.86048223007293 epoch‚7 /loss/FRt2/detection_loss‚1.7338341017994954 /loss/FRt3/assignment_loss‚2.332912149952679 2loss/SRqqt3/assignment_loss‚0.8101198606465648 1loss/FBt4/assignment_loss‚0.07447656254953472 1loss/SRqqt3/detection_loss‚2.0675710583241975 0loss/FRt4/assignment_loss‚3.4323334978974382 /loss/FRt1/detection_loss‚2.2733126941683808 /loss/FBt1/detection_loss‚2.4157486028141446 /loss/FBt2/detection_loss‚0.5570299926238429 _runtime‚10916.00056056 0loss/FRt1/assignment_loss‚2.0253814065710025 _step‚772 1loss/FBt1/assignment_loss‚0.18178917948130094 /loss/FRt3/detection_loss‚2.4981227028297215 /loss/FRt4/detection_loss‚3.1686470923216445 %loss/total_loss‚39.85553166430546 /loss/FBt3/detection_loss‚2.1849033653736116 0loss/FRt2/assignment_loss‚1.2316811717286402 2loss/SRqqt1/assignment_loss‚1.2724834349385257 1loss/FBt3/assignment_loss‚0.41187456910879516 .loss/FBt4/detection_loss‚2.873715857664744 trainer/global_step‚9399 2loss/SRqqt2/assignment_loss‚1.1527774334747598 % _timestamp‚1.7885212905304549e+09 2loss/SRqqt4/assignment_loss‚1.6463097362291246„µo]’²'‚(:‡ ÎÖêÔñ²Ò€ gpu.0.gpu‚17.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚164.372' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚41.093 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0„ šte³'‚(:Ú ÎÖêÔ„ȶ) disk./.usagePercent‚75.83668702489132 disk.sdb1.out‚0 disk.sda1.in‚0 disk.nvme0n1.in‚ 619.63671875 disk.sdb.in‚3.1875 disk.sda.in‚ 6.08203125% disk./.usageGB‚2712.2872619628906 disk.nvme1n1.in‚ 498.80078125 disk.dm-0.in‚ 1118.4375 disk.sda.out‚ 1250.703125 disk.sda2.out‚ 1250.703125) proc.memory.availableMB‚ 1293.37109375 cpu‚25.20145497174146 proc.cpu.threads‚17 disk.rbd0.out‚ 3.087890625 disk.sdb2.in‚3.1875 disk.sdb2.out‚ 1250.703125 disk.sdb.out‚ 1250.703125* proc.memory.percent‚18.320035934448242 disk.sdb1.in‚0 disk.md0.in‚ 9.26953125 disk.md0.out‚ 1205.58203125$ memory_percent‚92.10588932037354 disk.rbd2.in‚0 disk.sda1.out‚0! disk.dm-0.out‚119729.87890625 disk.md0p1.in‚ 9.26953125 disk.md0p1.out‚ 1205.58203125" disk.nvme0n1.out‚ 57845.2421875 disk.sda2.in‚ 6.08203125 network.sent‚ 2.4455334e+07 network.recv‚ 3.493412e+06" proc.memory.rssMB‚ 3001.5546875 disk.rbd0.in‚0 disk.rbd2.out‚ 3.03759765625# disk.nvme1n1.out‚61884.636718750]´'‚(:’ ÝÖêԊ›ÿ gpu.0.gpu‚52.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚117.81' gpu.0.enforcedPowerLimitWatts‚400.0) gpu.0.powerPercent‚29.452499999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¾ccµ'‚(:Ø ÝÖêÔî™ìÿ disk.sda.out‚ 1252.26953125 disk.sda2.out‚ 1252.26953125% disk./.usageGB‚2712.1962966918945 disk.md0p1.out‚ 1207.09375# memory_percent‚92.1032190322876" proc.memory.rssMB‚ 3001.5546875 disk.dm-0.out‚119791.6328125 disk.sdb.out‚ 1252.26953125 disk.sda2.in‚ 6.08203125 disk.rbd0.in‚0 disk.sda.in‚ 6.08203125 disk.rbd0.out‚ 3.087890625 cpu‚25.20123246973349 proc.cpu.threads‚17 network.recv‚ 3.498139e+06 disk.sdb1.out‚0 disk.rbd2.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-0.in‚ 1118.4375 disk.md0p1.in‚ 9.26953125 disk.nvme0n1.in‚ 619.63671875 disk.sdb1.in‚0" disk.nvme0n1.out‚ 57906.8046875 disk.nvme1n1.in‚ 498.80078125* proc.memory.percent‚18.320035934448242 network.sent‚ 2.4552397e+07 disk.md0.in‚ 9.26953125) disk./.usagePercent‚75.83414359784236 disk.sdb2.out‚ 1252.26953125! disk.nvme1n1.out‚ 61884.828125) proc.memory.availableMB‚ 1293.80859375 disk.md0.out‚ 1207.09375 disk.rbd2.out‚ 3.03759765625 disk.sdb2.in‚3.1875 disk.sdb.in‚3.1875{_’¶'‚(:‡ ìÖêÔ€”€ gpu.0.gpu‚45.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚37.0 gpu.0.powerWatts‚148.556' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚37.139 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0º@u·'‚(:ê ìÖêÔ†³¸€$ disk./.usageGB‚2712.364532470703" disk.nvme0n1.out‚ 58026.3046875! disk.sdb.out‚1253.63037109375 disk.rbd2.out‚ 3.03759765625 disk.sdb2.in‚3.1875 disk.nvme0n1.in‚ 619.63671875 disk.sda.in‚ 6.08203125 disk.rbd2.in‚0* proc.memory.percent‚18.320035934448242) disk./.usagePercent‚75.83884754062998 disk.dm-0.in‚ 1118.4375 disk.md0p1.in‚ 9.26953125" disk.sda2.out‚1253.63037109375# disk.nvme1n1.out‚61885.51953125 disk.md0.out‚ 1208.40234375% proc.memory.availableMB‚ 1293.5625" proc.memory.rssMB‚ 3001.5546875 disk.rbd0.out‚ 3.087890625 disk.sdb1.out‚0 disk.sdb.in‚3.1875" disk.sdb2.out‚1253.63037109375 disk.sda1.out‚0 network.sent‚ 2.4558405e+07 disk.rbd0.in‚0 disk.nvme1n1.in‚ 498.80078125 disk.sdb1.in‚0 disk.md0.in‚ 9.26953125 cpu‚25.201000149674822 network.recv‚ 3.502376e+06! disk.dm-0.out‚119911.82421875 disk.md0p1.out‚ 1208.40234375! disk.sda.out‚1253.63037109375 disk.sda1.in‚0 proc.cpu.threads‚17 disk.sda2.in‚ 6.08203125$ memory_percent‚92.10472106933594¯ECÏ’¸'‚(:‡ ûÖêÔó©²ÿ gpu.0.gpu‚53.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚171.168' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.792 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ìóD8¹'‚(:ç ûÖêÔ¿¨åÿ* proc.memory.percent‚18.32o*:0035934448242 disk.nvme0n1.in‚ 619.63671875! disk.sda.out‚1255.13818359375 disk.rbd0.out‚ 3.087890625 disk.rbd2.in‚0% disk./.usageGB‚2712.3714752197266 disk.sda1.in‚0 network.recv‚ 3.506663e+06) disk./.usagePercent‚75.83904166279845 disk.md0.out‚ 1209.859375" disk.sdb2.out‚1255.13818359375 disk.dm-0.in‚ 1118.4375 disk.md0p1.in‚ 9.26953125! disk.sdb.out‚1255.13818359375$ memory_percent‚92.10319519042969 disk.sda1.out‚0# disk.nvme0n1.out‚58102.66015625 disk.sdb.in‚3.1875 disk.rbd0.in‚0% proc.memory.availableMB‚ 1293.8125 proc.cpu.threads‚17 network.sent‚ 2.4562006e+07 disk.nvme1n1.in‚ 498.80078125 disk.sdb1.in‚0 disk.rbd2.out‚ 3.03759765625 disk.sdb2.in‚3.1875 disk.dm-0.out‚119988.3203125 disk.sda.in‚ 6.08203125" proc.memory.rssMB‚ 3001.5546875 cpu‚25.200774425042905 disk.sda2.in‚ 6.08203125" disk.sda2.out‚1255.13818359375# disk.nvme1n1.out‚61885.66015625 disk.sdb1.out‚0 disk.md0.in‚ 9.26953125 disk.md0p1.out‚ 1209.859375<ƒÃ˜4º'/ _wandbruntime‚10971  _runtime‚10971÷Ҿɻ'à ' lr-AdamW/pg2‚0.00011874874141926036 trainer/global_step‚9449 % _timestamp‚1.7885213457354615e+09 _runtime‚10971.20597674 _step‚773 ' lr-AdamW/pg1‚0.00011874874141926036ÚíÖ:μ'È % _timestamp‚1.7885213457354615e+09 _runtime‚10971.20597674 _step‚773 ' lr-AdamW/pg1‚0.00011874874141926036 ' lr-AdamW/pg2‚0.00011874874141926036 trainer/global_step‚9449…Sèƒ4½'/ _wandbruntime‚10971  _runtime‚10971¸S›(¾'P G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 194783472 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚769576887861946 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 236917162 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 401960960 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411420 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 121122293 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1187 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10734831967232 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 431700634 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955591168 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007241 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917410816 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955591168 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 % _timestamp‚1.7885213457366965e+09 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚776156647427584 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚9449 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚611 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1212 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚776156629424640 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚879752174773760 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚56628539 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11212253357568 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214665 _step‚774 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 194783472 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚879752429670912 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚868540176313344 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚786890561984000 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚776156629424640 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407872 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 _runtime‚10971.207632367 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26088 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26088 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 236891074 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177047040 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 121122295 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10733932559360 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410816 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10693914203662 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10694811615082 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚776156647427584 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚868540152373248 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 236891074 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 230957056 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26090 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10733932559360 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚769575973411166 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚786891479394816 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚786890561984000 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407872 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 æUÑ (¿'†P M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 236891074 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177047040 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 121122295 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10733932559360 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410816 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10693914203662 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10694811615082 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚776156647427584 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚868540152373248 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 236891074 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 230957056 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26090 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10733932559360 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚769575973411166 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚786891479394816 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚786890561984000 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407872 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 194783472 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚769576887861946 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 236917162 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 401960960 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411420 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 121122293 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1187 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10734831967232 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 431700634 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955591168 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007241 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917410816 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955591168 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 % _timestamp‚1.7885213457366965e+09 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚776156647427584 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚9449 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚611 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1212 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚776156629424640 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚879752174773760 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚56628539 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11212253357568 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214665 _step‚774 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 194783472 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚879752429670912 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚868540176313344 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚786890561984000 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚776156629424640 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407872 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 _runtime‚10971.207632367 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26088 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26088 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1†>7V»4À'/ _wandbruntime‚10972  _runtime‚10972ŒVºt'Á'‹N K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 194803368 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚786971305082880 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26114 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 236941779 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450988 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11213200916992 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚776236259720192 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177047040 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 177768900 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚769655062166612 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 431719033 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚9449 _runtime‚10972.391909833 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 431745149 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 121134671 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚879842672510976 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10695920276608 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10735045362688 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 431745149 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899421184 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917424128 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31686 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚611 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955591168 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10735045362688 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 194803370 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 194803370 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 194803368 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 121134669 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 % _timestamp‚1.7885213469193623e+09 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚769655976617600 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 236941779 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚613 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11213417180672 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 236915665 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0Xš“C)Â'N ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚786971305082880 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26114 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 SEDevicTØŽ}í$eStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 236941779 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450988 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11213200916992 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚776236259720192 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177047040 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 177768900 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚769655062166612 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 431719033 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚9449 _runtime‚10972.391909833 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 431745149 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 121134671 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚879842672510976 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10695920276608 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10735045362688 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 431745149 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899421184 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917424128 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31686 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚611 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955591168 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10735045362688 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 194803370 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 194803370 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 194803368 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 121134669 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 % _timestamp‚1.7885213469193623e+09 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚769655976617600 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 236941779 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚613 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11213417180672 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 236915665 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 194803368 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712‡Ê%d>4Ã'/ _wandbruntime‚10972  _runtime‚10972 ø\Ä'Ö &loss/total_loss‚37.851657364496134 0loss/FRt1/assignment_loss‚2.6855462988217673 2loss/SRqqt1/assignment_loss‚1.2320593259538766 2loss/SRqqt4/assignment_loss‚1.6560058711289332 1loss/FBt2/assignment_loss‚0.09118788118526058 /loss/FRt1/detection_loss‚2.0077087649325325 /loss/FRt2/detection_loss‚1.6111663087959522 0loss/SRqqt1/detection_loss‚1.417266261867351 2loss/SRqqt2/assignment_loss‚0.9331450237245378 /loss/FBt3/assignment_loss‚0.089128407484471 1loss/SRqqt4/detection_loss‚2.3787127106354156 /loss/FBt4/detection_loss‚2.7015564858913423 /loss/FRt4/assignment_loss‚3.109476990169949 /loss/FRt3/detection_loss‚2.5672707822587757 % _timestamp‚1.7885213469230154e+09 0loss/FBt1/assignment_loss‚0.0538025389916064 /loss/FBt1/detection_loss‚2.5404859951564243 0loss/FRt3/assignment_loss‚1.9904722425672743 1loss/SRqqt3/assignment_loss‚1.174332389365072 /loss/FBt3/detection_loss‚2.5507043136490717 _step‚776 epoch‚7 0loss/FRt2/assignment_loss‚0.7006145417690277 /loss/FBt2/detection_loss‚0.8116351202359959 1loss/FBt4/assignment_loss‚0.15535273774216876 -loss/FRt4/detection_loss‚2.79406898882654 trainer/global_step‚9449 0loss/SRqqt2/detection_loss‚0.998542369043423 1loss/SRqqt3/detection_loss‚1.6014150142993615 _runtime‚10972.393204649„Ü;aÅ'Û 1loss/FBt4/assignment_loss‚0.15535273774216876 -loss/FRt4/detection_loss‚2.79406898882654 trainer/global_step‚9449 0loss/SRqqt2/detection_loss‚0.998542369043423 1loss/SRqqt3/detection_loss‚1.6014150142993615 _runtime‚10972.393204649 &loss/total_loss‚37.851657364496134 0loss/FRt1/assignment_loss‚2.6855462988217673 2loss/SRqqt1/assignment_loss‚1.2320593259538766 2loss/SRqqt4/assignment_loss‚1.6560058711289332 1loss/FBt2/assignment_loss‚0.09118788118526058 /loss/FRt1/detection_loss‚2.0077087649325325 /loss/FRt2/detection_loss‚1.6111663087959522 0loss/SRqqt1/detection_loss‚1.417266261867351 2loss/SRqqt2/assignment_loss‚0.9331450237245378 /loss/FBt3/assignment_loss‚0.089128407484471 1loss/SRqqt4/detection_loss‚2.3787127106354156 /loss/FBt4/detection_loss‚2.7015564858913423 /loss/FRt4/assignment_loss‚3.109476990169949 /loss/FRt3/detection_loss‚2.5672707822587757 % _timestamp‚1.7885213469230154e+09 0loss/FBt1/assignment_loss‚0.0538025389916064 /loss/FBt1/detection_loss‚2.5404859951564243 0loss/FRt3/assignment_loss‚1.9904722425672743 1loss/SRqqt3/assignment_loss‚1.174332389365072 /loss/FBt3/detection_loss‚2.5507043136490717 _step‚776 epoch‚7 0loss/FRt2/assignment_loss‚0.7006145417690277 /loss/FBt2/detection_loss‚0.8116351202359959ˆ‹1q%“Æ'‚(:ˆ Š×êÔó°Ïÿ gpu.0.gpu‚54.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚167.434' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚41.8585 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÇÏ tÇ'‚(:é Š×êÔÀ²Ñ‚" proc.memory.rssMB‚ 3001.5546875$ disk./.usageGB‚2712.747905731201 disk.dm-0.out‚120037.7265625 disk.sda2.in‚ 6.08203125 disk.rbd0.in‚0$ memory_percent‚92.10727214813232" disk.nvme1n1.out‚ 61885.9296875" disk.sdb2.out‚1257.15185546875 disk.dm-0.in‚ 1118.4375! disk.sdb.out‚1257.15185546875 disk.sda.in‚ 6.08203125 disk.md0.in‚ 9.26953125 proc.cpu.threads‚17 disk.sda1.out‚0 disk.sdb.in‚3.1875" disk.sda2.out‚1257.15185546875 disk.nvme1n1.in‚ 498.80078125* proc.memory.percent‚18.320035934448242 disk.sdb2.in‚3.1875! disk.nvme0n1.out‚ 58151.796875) proc.memory.availableMB‚ 1293.14453125 disk.sdb1.out‚0 disk.md0p1.in‚ 9.26953125( disk./.usagePercent‚75.8495668174477 disk.rbd2.in‚0 disk.rbd2.out‚ 3.03759765625 disk.md0.out‚ 1211.81640625 cpu‚25.200509833249235 network.recv‚ 3.511264e+06 disk.md0p1.out‚ 1211.81640625 disk.nvme0n1.in‚ 619.63671875 disk.rbd0.out‚ 3.087890625 disk.sdb1.in‚0 network.sent‚ 2.458118e+07 disk.sda1.in‚0! disk.sda.out‚1257.15185546875&©=‰”È'‚(:‰ ™×êÔкÇÿ gpu.0.gpu‚56.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚131.173' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚32.79325 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0òŠv‡mÉ'‚(:â ™×êÔ«ýÿ disk.rbd0.in‚0) disk./.usagePercent‚75.86010082492109 disk.sdb1.in‚0 disk.md0p1.out‚ 1213.40234375# proc.memory.availableMB‚1293.25" proc.memory.rssMB‚ 3001.5546875 cpu‚25.200295040441667 disk.rbd2.in‚0 disk.dm-0.in‚ 1118.4375 disk.md0p1.in‚ 9.26953125 disk.sdb.out‚1258.7939453125 disk.sda.in‚ 6.08203125* proc.memory.percent‚18.320035934448242! disk.sdb2.out‚1258.7939453125 network.sent‚ 2.4675318e+07 disk.md0.in‚ 9.26953125! disk.dm-0.out‚120037.96484375 disk.nvme0n1.in‚ 619.63671875! disk.nvme0n1.out‚ 58151.859375 disk.sdb.in‚3.1875! disk.sda2.out‚1258.7939453125 disk.nvme1n1.in‚ 498.80078125 disk.rbd0.out‚ 3.087890625 network.recv‚ 3.51631e+06 disk.sdb1.out‚0 disk.sda.out‚1258.7939453125$ memory_percent‚92.10662841796875 disk.sda1.out‚0 disk.sdb2.in‚3.1875# disk.nvme1n1.out‚61886.10546875 disk.md0.out‚ 1213.40234375 disk.sda1.in‚0 disk.sda2.in‚ 6.08203125 proc.cpu.threads‚17$ disk./.usageGB‚2713.124652862549 disk.rbd2.out‚ 3.10009765625€’ñ’Ê'‚(:‡ ¨×êÔ¦ƒ¶ÿ gpu.0.gpu‚2.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚36.0 gpu.0.powerWatts‚161.254' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚40.3135 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ô˜ÒrË'‚(:ç ¨×êÔü¹Ü€ network.recv‚ 3.520482e+06% disk./.usageGB‚2713.3726348876953$ memory_percent‚92.10505485534668" proc.memory.rssMB‚ 3001.5546875 disk.sdb.in‚3.1875! disk.sda.out‚1260.23193359375 disk.sda2.in‚ 6.08203125 disk.rbd0.in‚0 disk.md0.in‚ 9.26953125! disk.sdb.out‚1260.23193359375 disk.sda.in‚ 6.08203125 disk.md0.out‚ 1214.7734375( proc.memory.availableMB‚ 1293.5078125 cpu‚25.20006457322663 disk.sda1.in‚0 disk.rbd0.out‚ 3.087890625" disk.nvme1n1.out‚ 61886.6953125* proc.memory.percent‚18.320035934448242 disk.md0p1.out‚ 1214.7734375 disk.nvme0n1.in‚ 619.63671875" disk.sda2.out‚1260.23193359375 proc.cpu.threads‚17) disk./.usagePercent‚75.86703450613273 disk.md0p1.in‚ 9.26953125" disk.nvme0n1.out‚ 58770.7734375 disk.nvme1n1.in‚ 498.80078125 disk.rbd2.in‚0 disk.rbd2.out‚ 3.10009765625" disk.sdb2.out‚1260.23193359375 network.sent‚ 2.4683049e+07 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.sdb2.in‚3.1875 disk.sda1.out‚0 disk.dm-0.in‚ 1118.4375 disk.dm-0.out‚ 120657.46875I”Ì'‚(:‰ ·×êÔ°ºŒ gpu.0.gpu‚48.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚169.161' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.29025 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÄZñjÍ'‚(:ß ·×êÔ¸ªƒ$ disk./.usageGB‚2713.873634338379! disk.sda2.out‚1262.1689453125! disk.nvme1n1.out‚ 61886.828125 disk.sda1.out‚0 disk.sda.out‚1262.1689453125 disk.sda1.in‚0 disk.dm-0.out‚ 120765.421875) proc.memory.availableMB‚ 1293.75390625 cpu‚25.199766103869212 disk.rbd0.in‚0 disk.rbd2.in‚0 disk.nvme0n1.in‚ 619.63671875" proc.memory.rssMB‚ 3001.5546875 network.recv‚ 3.525225e+06) disk./.usagePercent‚75.88104266045839 disk.sda.in‚ 6.08203125 disk.md0p1.out‚ 1216.640625 disk.md0p1.in‚ 9.26953125 disk.sdb1.in‚0 disk.md0.in‚ 9.26953125 disk.md0.out‚ 1216.640625 disk.dm-0.in‚ 1118.4375 disk.sdb.out‚1262.1689453125 disk.rbd2.out‚ 3.1240234375! disk.sdb2.out‚1262.1689453125* proc.memory.percent‚18.320035934448242 network.sent‚ 2.4687151e+07 disk.sda2.in‚ 6.08203125 disk.nvme1n1.in‚ 498.80078125 disk.sdb1.out‚0$ memory_percent‚92.10355281829834 disk.sdb.in‚3.1875 disk.rbd0.out‚ 3.087890625 disk.sdb2.in‚3.1875 disk.nvme0n1.out‚ 58878.59375 proc.cpu.threads‚17ååž74Î'/ _wandbruntime‚11028  _runtime‚11028^V.òÈÏ' & lr-AdamW/pg1‚0.0001145048483193893 & lr-AdamW/pg2‚0.0001145048483193893 trainer/global_step‚9499 % _timestamp‚1.7885214031580265e+09 _runtime‚11028.628396779 _step‚777mLs‡ÍÐ'Ç & lr-AdamW/pg2‚0.0001145048483193893 trainer/global_step‚9499 % _timestamp‚1.7885214031580265e+09 _runtime‚11028.628396779 _step‚777 & lr-AdamW/pg1‚0.0001145048483193893‰ù·Þ4Ñ'/ _wandbruntime‚11028  _runtime‚11028ɸõ'(Ò'P UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 195778370 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚790927731341824 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚780138144202240 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 195778372 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10750258272190 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450780 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 121741095 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚884276603162624 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚629 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26090 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚780138162205184 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚780138144202240 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917410816 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26088 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410816 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚873006113410048 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10749360860770 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 178657017 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1212 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚790928648752640 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 238123976 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214665 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚873006137350144 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 433902346 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407872 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26088 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10789587139584 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 238150064 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 _runtime‚11028.629503534 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10790486547456 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚762781063913904 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚884276360848384 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 401960960 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10789587139584 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955591168 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11270247438336 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 178657648 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 242314240 trainer/global_step‚9499 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10790486547456 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚56915924 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 % _timestamp‚1.7885214031590745e+09 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11270465812480 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411420 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007241 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171OƼf– Ó'†P UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 195778372 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10750258272190 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450780 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 121741095 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚884276603162624 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚629 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26090 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚780138162205184 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚780138144202240 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917410816 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26088 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410816 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚873006113410048 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10749360860770 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 178657017 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1212 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚790928648752640 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 238123976 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214665 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚873006137350144 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.lar¨9Þvge_pool.current‚23940096 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 433902346 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407872 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26088 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10789587139584 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 238150064 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 _runtime‚11028.629503534 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10790486547456 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚762781063913904 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚884276360848384 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 401960960 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10789587139584 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955591168 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11270247438336 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 178657648 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 242314240 trainer/global_step‚9499 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10790486547456 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚56915924 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 % _timestamp‚1.7885214031590745e+09 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11270465812480 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411420 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007241 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 195778370 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚790927731341824 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚780138144202240 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090Š¬5ùb4Ô'/ _wandbruntime‚11029  _runtime‚11029‡Ž'Õ'‹N K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 195798268 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 195798268 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 178675715 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11271411092480 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 433946835 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 433946835 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚884366843723264 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚873095432630784 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 _step‚779 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚56922244 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899421184 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 trainer/global_step‚9499 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚780217792500736 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10750469522088 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007241 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚780217774497792 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1187 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177047040 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 238174681 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11271633647616 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411628 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚791009391864832 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10751366933716 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 433972951 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 238148567 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10791599364096 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚634 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚873095456570880 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31686 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚56921612 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 433972951 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚632 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 238174681 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26114 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚791008474440704 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917424128 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10791599364096 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26114 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚884367090218496 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚762859044008032 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10790699942912 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 238148567 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 121753469 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10790699942912 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 178675081 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚773609513530120 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411628 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚791009391864832 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10751366933716 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 433972951 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 238148567 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10791599364096 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚634 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚873095456570880 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31686 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚56921612 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 433972951 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚632 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 238174681 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26114 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚791008474440704 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917424128 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10791599364096 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26114 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚884367090218496 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚762859044008032 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10790699942912 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 238148567 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 121753469 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10790699942912 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 178675081 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚773609513530120 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 trainer/global_step‚9499 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚780217792500736 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10750469522088 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007241 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚780217774497792 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1187 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177047040 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 238174681‹X'Ëç4×'/ _wandbruntime‚11029  _runtime‚11029näÂ^Ø'Ø $ _timestamp‚1.788521404136827e+09 2loss/SRqqt2/assignment_loss‚0.7702398556769671 .loss/FRt3/detection_loss‚2.411314611323178 /loss/FRt4/detection_loss‚2.8924675153361425 .loss/FBt1/detection_loss‚2.310717591217586 /loss/FBt3/detection_loss‚2.4997322857379913 _step‚780 /loss/FRt2/detection_loss‚1.8093354188371449 /loss/FRt1/assignment_loss‚2.498012679815292 2loss/SRqqt1/assignment_loss‚1.0436254269443452 2loss/SRqqt3/assignment_loss‚0.9759829763286546 1loss/FBt2/assignment_loss‚0.11799411153715453 1loss/SRqqt1/detection_loss‚1.4569552419086298 %loss/total_loss‚36.95517090762966 _runtime‚11029.606991082 0loss/FRt3/assignment_loss‚1.9635395228862762 1loss/SRqqt4/detection_loss‚1.9794243929847595 /loss/FBt2/detection_loss‚0.5633335622154515 0loss/FRt4/assignment_loss‚3.1023781652803772 2loss/SRqqt4/assignment_loss‚1.6331494527478372 1loss/FBt1/assignment_loss‚0.06756230521049085 .loss/FRt1/detection_loss‚1.885228221224388 1loss/SRqqt3/detection_loss‚1.7248641091818904 epoch‚7 1loss/SRqqt2/detection_loss‚0.8791607612640493 0loss/FRt2/assignment_loss‚1.1873140171170236 /loss/FBt4/detection_loss‚2.6621367633342743 1loss/FBt4/assignment_loss‚0.25526505638254093 0loss/FBt3/assignment_loss‚0.2654368631372108 trainer/global_step‚9499Ä]cÙ'Ý 1loss/FBt2/assignment_loss‚0.11799411153715453 1loss/SRqqt1/detection_loss‚1.4569552419086298 %loss/total_loss‚36.95517090762966 _runtime‚11029.606991082 0loss/FRt3/assignment_loss‚1.9635395228862762 1loss/SRqqt4/detection_loss‚1.9794243929847595 /loss/FBt2/detection_loss‚0.5633335622154515 0loss/FRt4/assignment_loss‚3.1023781652803772 2loss/SRqqt4/assignment_loss‚1.6331494527478372 1loss/FBt1/assignment_loss‚0.06756230521049085 .loss/FRt1/detection_loss‚1.885228221224388 1loss/SRqqt3/detection_loss‚1.7248641091818904 epoch‚7 1loss/SRqqt2/detection_loss‚0.8791607612640493 0loss/FRt2/assignment_loss‚1.1873140171170236 /loss/FBt4/detection_loss‚2.6621367633342743 1loss/FBt4/assignment_loss‚0.25526505638254093 0loss/FBt3/assignment_loss‚0.2654368631372108 trainer/global_step‚9499 $ _timestamp‚1.788521404136827e+09 2loss/SRqqt2/assignment_loss‚0.7702398556769671 .loss/FRt3/detection_loss‚2.411314611323178 /loss/FRt4/detection_loss‚2.8924675153361425 .loss/FBt1/detection_loss‚2.310717591217586 /loss/FBt3/detection_loss‚2.4997322857379913 _step‚780 /loss/FRt2/detection_loss‚1.8093354188371449 /loss/FRt1/assignment_loss‚2.498012679815292 2loss/SRqqt1/assignment_loss‚1.0436254269443452 2loss/SRqqt3/assignment_loss‚0.9759829763286546Œõ¹ü\’Ú'‚(:‡ Æ×êÔôóßý gpu.0.gpu‚56.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚37.0 gpu.0.powerWatts‚128.056' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚32.014 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0€<÷pÛ'‚(:å Æ×êÔ›Óôþ disk.dm-0.out‚121171.5859375 disk.sda.in‚ 6.08203125 disk.md0.in‚ 9.26953125 disk.md0p1.out‚ 1217.98046875 disk.nvme0n1.in‚ 619.63671875 disk.sdb.in‚3.1875# memory_percent‚92.1051025390625 proc.cpu.threads‚17! disk.sda.out‚1263.55712890625 disk.sda2.in‚ 6.08203125 disk.rbd0.in‚0 disk.sda1.out‚0 disk.sda1.in‚0" proc.memory.availableMB‚1293.5" disk.sda2.out‚1263.55712890625 disk.nvme1n1.in‚ 498.80078125 disk.sdb2.in‚3.1875 disk.md0p1.in‚ 9.26953125* proc.memory.percent‚18.320035934448242 cpu‚25.19950817085009% disk./.usageGB‚2713.8716163635254 disk.md0.out‚ 1217.98046875 disk.dm-0.in‚ 1118.4375# disk.nvme0n1.out‚59284.52734375" proc.memory.rssMB‚ 3001.5546875 disk.sdb1.out‚0 network.recv‚ 3.529902e+06 disk.sdb1.in‚0 disk.rbd2.in‚0 disk.rbd2.out‚ 3.1240234375 network.sent‚ 2.4693414e+07) disk./.usagePercent‚75.88098623703691# disk.nvme1n1.out‚61887.05859375! disk.sdb.out‚1263.55712890625 disk.rbd0.out‚ 3.087890625" disk.sdb2.out‚1263.55712890625ü?$z”Ü'‚(:‰ Õ×êÔ„¨ÿ gpu.0.gpu‚27.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚37.0 gpu.0.powerWatts‚116.583' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚29.14575 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0™¡·WÝ'‚(:ã Õ×êÔˈùÿ network.sent‚ 2.4802536e+07 disk.nvme0n1.in‚ 619.63671875 disk.sdb.in‚3.1875! disk.dm-0.out‚121400.83984375 network.recv‚ 3.534629e+06 disk.sda.in‚ 6.08203125 disk.rbd0.in‚0 disk.rbd2.out‚ 3.18701171875 disk.sdb2.out‚1265.087890625 disk.sda1.in‚0$ disk./.usageGB‚2713.871639251709 disk.md0p1.in‚ 9.26953125 disk.md0p1.out‚ 1219.44921875 disk.rbd0.out‚ 3.087890625$ memory_percent‚92.10524559020996" proc.memory.rssMB‚ 3001.5546875 proc.cpu.threads‚17" disk.nvme0n1.out‚ 59513.6484375 disk.sdb.out‚1265.087890625* proc.memory.percent‚18.320035934448242 cpu‚25.19930084041412 disk.sda2.out‚1265.087890625 disk.dm-0.in‚ 1118.4375( disk./.usagePercent‚75.8809868770001 disk.sda.out‚1265.087890625# disk.nvme1n1.out‚61887.19140625 disk.sdb1.out‚0 disk.sda1.out‚0 disk.sda2.in‚ 6.08203125 disk.md0.in‚ 9.26953125 disk.rbd2.in‚0( proc.memory.availableMB‚ 1293.4765625 disk.nvme1n1.in‚ 498.80078125 disk.sdb1.in‚0 disk.md0.out‚ 1219.44921875å@1T disk.sdb2.in‚3.1875”®4’Þ'‚(:‡ ä×êÔÑüÀ gpu.0.gpu‚54.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚169.61' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.4025 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0&¶T=kß'‚(:à ä×êÔËãÚ‚" proc.memory.rssMB‚ 3001.5546875 disk.sdb2.in‚3.1875 disk.sda1.in‚0" disk.nvme0n1.out‚ 59666.6328125! disk.sdb.out‚1266.51123046875) proc.memory.availableMB‚ 1293.67578125* proc.memory.percent‚18.320035934448242 disk.rbd2.out‚ 3.18701171875 network.sent‚ 2.481023e+07 disk.rbd0.in‚0 disk.md0.out‚ 1220.8125 disk.md0p1.out‚ 1220.8125 proc.cpu.threads‚17 disk.sda1.out‚0 network.recv‚ 3.546228e+06 disk.md0.in‚ 9.26953125 disk.nvme0n1.in‚ 619.63671875" disk.sda2.out‚1266.51123046875 cpu‚25.19908593713949% disk./.usageGB‚2713.8712768554688 disk.nvme1n1.in‚ 498.80078125 disk.dm-0.out‚ 121554.0625 disk.sda.in‚ 6.08203125) disk./.usagePercent‚75.88097674424955 disk.rbd0.out‚ 3.087890625 disk.sdb1.in‚0 disk.md0p1.in‚ 9.26953125 disk.sda2.in‚ 6.08203125" disk.nvme1n1.out‚ 61887.4296875 disk.sdb1.out‚0" disk.sdb2.out‚1266.51123046875 disk.dm-0.in‚ 1118.4375$ memory_percent‚92.10402965545654 disk.rbd2.in‚0 disk.sdb.in‚3.1875! disk.sda.out‚1266.51123046875x >f4à'/ _wandbruntime‚11082  _runtime‚11082+ò6Êá'Ä _runtime‚11082.948147802 _step‚781 ' lr-AdamW/pg1‚0.00011032646434053869 ' lr-AdamW/pg2‚0.00011032646434053869 trainer/global_step‚9549 % _timestamp‚1.7885214574776838e+09«JÐ!Ïâ'É ' lr-AdamW/pg2‚0.00011032646434053869 trainer/global_step‚9549 % _timestamp‚1.7885214574776838e+09 _runtime‚11082.948147802 _step‚781 ' lr-AdamW/pg1‚0.00011032646434053869Œ² ã4ã'/ _wandbruntime‚11082  _runtime‚11082ÁtX(ä'P QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450780 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 % _timestamp‚1.7885214574787781e+09 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955591168 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚794964900699648 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 179563133 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955591168 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 230957056 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214665 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26088 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _runtime‚11082.949526619 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 254897152 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10804807517878 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚794965818110464 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 196773270 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚888800653251072 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10805704929298 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007241 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917410816 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411420 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 196773272 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚794964900699648 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚601 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 122359895 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚57203839 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 401960960 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 _step‚782 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10846141127680 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10845241719808 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410816 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 179563734 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1212 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚877472098386944 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚784119676982784 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 196773272 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚794964900699648 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚601 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 122359895 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚57203839 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 401960960 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 _step‚782 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10846141127680 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10845241719808 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410816 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 179563734 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1212 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚877472098386944 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚784119676982784 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _runtime‚11082.949526619 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 254897152 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10804807517878 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚794965818110464 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 196773270 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚888800653251072 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10805704929298 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007241 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917410816 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411420 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 JDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 179581147 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚795045643798528 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 122372269 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899421184 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 436200753 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1187 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 248592384 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚795046561222656 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955591168 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10847253944320 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚617 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 trainer/global_step‚9549 % _timestamp‚1.7885214585725288e+09 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 239381469 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 122372271 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917424128 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚57209493 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10806813590824 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31686 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚877561417607680 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚777563964893628 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚888890882333696 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 401960960 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 436200753 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26114 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007241 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10847253944320 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171T5C?è'N OADeviceStatsMonitor.on_train_batch_end/allocated_bytesæ›Y×&.all.current‚ 917424128 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚57209493 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10806813590824 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31686 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚877561417607680 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚777563964893628 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚888890882333696 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 401960960 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 436200753 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26114 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007241 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10847253944320 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 _runtime‚11084.043321244 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 239407583 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 196793168 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚795046561222656 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚784199307278336 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 179581147 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚795045643798528 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 122372269 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899421184 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 436200753 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1187 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 248592384 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚795046561222656 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955591168 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10847253944320 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚617 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 trainer/global_step‚9549 % _timestamp‚1.7885214585725288e+09 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 239381469 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 122372271 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564ÅpT4é'/ _wandbruntime‚11084  _runtime‚11084Šèþ`ê'Ú /loss/FRt3/assignment_loss‚2.466826070296137 /loss/FRt2/detection_loss‚1.6790214031583295 1loss/SRqqt4/detection_loss‚2.0846107623406818 /loss/FBt4/detection_loss‚2.7139879117409387 0loss/FRt2/assignment_loss‚1.2319201725833822 1loss/FBt2/assignment_loss‚0.12212200424900514 /loss/FBt1/detection_loss‚2.1113401597196404 0loss/FBt1/assignment_loss‚0.0720024436886888 /loss/FBt2/detection_loss‚0.6503426280279441 2loss/SRqqt1/assignment_loss‚1.4462385946525294 _step‚784 % _timestamp‚1.7885214585750446e+09 2loss/SRqqt3/assignment_loss‚1.0851736681754975 2loss/SRqqt4/assignment_loss‚1.5876720151198762 _runtime‚11084.045184572 epoch‚7 trainer/global_step‚9549 0loss/FRt1/assignment_loss‚2.8004317492769477 /loss/FRt1/detection_loss‚1.9020581804514605 .loss/FRt3/detection_loss‚2.546987782379514 1loss/SRqqt2/detection_loss‚1.2258685643412173 1loss/SRqqt3/detection_loss‚1.5248842942071896 /loss/FBt3/detection_loss‚2.1466734317632823 &loss/total_loss‚38.760051971491215 0loss/FRt4/assignment_loss‚2.8399609327316284 1loss/FBt3/assignment_loss‚0.14343691137047734 0loss/FBt4/assignment_loss‚0.3439373036624905 .loss/FRt4/detection_loss‚3.520267489709352 1loss/SRqqt1/detection_loss‚1.5474779071253644 2loss/SRqqt2/assignment_loss‚0.9668095907196402B»8eë'ß 0loss/FBt1/assignment_loss‚0.0720024436886888 /loss/FBt2/detection_loss‚0.6503426280279441 2loss/SRqqt1/assignment_loss‚1.4462385946525294 _step‚784 % _timestamp‚1.7885214585750446e+09 2loss/SRqqt3/assignment_loss‚1.0851736681754975 2loss/SRqqt4/assignment_loss‚1.5876720151198762 _runtime‚11084.045184572 epoch‚7 trainer/global_step‚9549 0loss/FRt1/assignment_loss‚2.8004317492769477 /loss/FRt1/detection_loss‚1.9020581804514605 .loss/FRt3/detection_loss‚2.546987782379514 1loss/SRqqt2/detection_loss‚1.2258685643412173 1loss/SRqqt3/detection_loss‚1.5248842942071896 /loss/FBt3/detection_loss‚2.1466734317632823 &loss/total_loss‚38.760051971491215 0loss/FRt4/assignment_loss‚2.8399609327316284 1loss/FBt3/assignment_loss‚0.14343691137047734 0loss/FBt4/assignment_loss‚0.3439373036624905 .loss/FRt4/detection_loss‚3.520267489709352 1loss/SRqqt1/detection_loss‚1.5474779071253644 2loss/SRqqt2/assignment_loss‚0.9668095907196402 /loss/FRt3/assignment_loss‚2.466826070296137 /loss/FRt2/detection_loss‚1.6790214031583295 1loss/SRqqt4/detection_loss‚2.0846107623406818 /loss/FBt4/detection_loss‚2.7139879117409387 0loss/FRt2/assignment_loss‚1.2319201725833822 1loss/FBt2/assignment_loss‚0.12212200424900514 /loss/FBt1/detection_loss‚2.1113401597196404äÎIÇžì'‚(:“ ó×êÔáòŸ€ gpu.0.gpu‚56.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚37.0 gpu.0.powerWatts‚122.818' gpu.0.enforcedPowerLimitWatts‚400.0) gpu.0.powerPercent‚30.704500000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0µ°ç3kí'‚(:à ó×êÔçÙ€ disk.sda.out‚1268.4892578125 disk.sdb1.out‚0 disk.rbd0.out‚ 3.087890625" disk.nvme1n1.out‚ 61887.5546875 cpu‚25.198849214561264) disk./.usagePercent‚75.86617834204138! disk.sdb2.out‚1268.4892578125 disk.sdb.out‚1268.4892578125 disk.rbd0.in‚0 disk.rbd2.in‚0 disk.rbd2.out‚3.25 proc.cpu.threads‚17$ disk./.usageGB‚2713.342014312744 disk.nvme0n1.in‚ 619.63671875 disk.sdb.in‚3.1875 network.recv‚ 3.558161e+06 disk.dm-0.in‚ 1118.4375! disk.dm-0.out‚121649.98046875# disk.nvme0n1.out‚59762.42578125 disk.nvme1n1.in‚ 498.80078125 disk.md0.out‚ 1222.72265625" proc.memory.rssMB‚ 3001.5546875* proc.memory.percent‚18.320035934448242 disk.sda.in‚ 6.08203125! disk.sda2.out‚1268.4892578125$ memory_percent‚92.10684299468994 network.sent‚ 2.483311e+07 disk.sdb2.in‚3.1875 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.in‚ 6.08203125) proc.memory.availableMB‚ 1293.21484375 disk.md0p1.in‚ 9.26953125 disk.md0p1.out‚ 1222.72265625 disk.sdb1.in‚0 disk.md0.in‚ 9.26953125ÑÛýN’î'‚(:‡ ‚ØêÔÔøþ gpu.0.gpu‚43.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚173.344' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.336 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0uá¦(mï'‚(:â ‚ØêÔö†¡ÿ disk.rbd0.out‚ 3.087890625" disk.sdb2.out‚1269.89404296875 disk.dm-0.in‚ 1118.4375 disk.nvme0n1.in‚ 619.63671875 disk.rbd2.out‚3.25 disk.sdb2.in‚3.1875) proc.memory.availableMB‚ 1293.28515625$ disk./.usageGB‚2713.341655731201 disk.md0.in‚ 9.26953125 disk.sda1.in‚0 disk.sda1.out‚0* proc.memory.percent‚18.320035934448242 proc.cpu.threads‚17) disk./.usagePercent‚75.86616831595137! disk.sda.out‚1269.89404296875 disk.sda2.in‚ 6.08203125 disk.sdb1.in‚0 disk.rbd2.in‚0" proc.memory.rssMB‚ 3001.5546875 network.sent‚ 2.4931168e+07" disk.sda2.out‚1269.89404296875 disk.sdb1.out‚0 disk.md0p1.out‚ 1224.08203125$ memory_percent‚92.10641384124756 disk.rbd0.in‚0! disk.dm-0.out‚122017.77734375 disk.sda.in‚ 6.08203125 disk.nvme1n1.in‚ 498.80078125 disk.md0.out‚ 1224.08203125 disk.md0p1.in‚ 9.26953125# disk.nvme0n1.out‚60176.46484375 disk.sdb.in‚3.1875 network.recv‚ 3.564613e+06 disk.nvme1n1.out‚ 61887.8125! disk.sdb.out‚1269.89404296875 cpu‚25.198655292221463œTW ”ð'‚(:‰ ‘ØêÔõÑŸÿ gpu.0.gpu‚55.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚170.559' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.63975 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0O/Õ»Xñ'‚(:Í ‘ØêÔ¿Åóÿ disk.sda2.in‚ 6.08203125 disk.sdb2.in‚3.1875 disk.sdb1.in‚0 disk.sda1.out‚0 disk.dm-0.in‚ 1118.4375 proc.cpu.threads‚17 network.recv‚ 3.567985e+06 disk.md0.in‚ 9.26953125 disk.md0.out‚ 1225.546875" proc.memory.rssMB‚ 3001.5546875* proc.memory.percent‚18.320035934448242 network.sent‚ 2.4936364e+07 disk.rbd2.in‚0$ memory_percent‚92.10636615753174! disk.dm-0.out‚122078.66015625 disk.sdb.in‚3.1875 disk.sdb.out‚ 1271.421875) proc.memory.availableMB‚ 1293.29296875 disk.sda2.out‚ 1271.421875) disk./.usagePercent‚75.86694128482765$ disk./.usageGB‚2713.369300842285# disk.nvme1n1.out‚61887.99609375 disk.rbd2.out‚3.25 disk.sda.in‚ 6.08203125 disk.sdb2.out‚ 1271.421875 disk.md0p1.out‚ 1225.546875 disk.nvme0n1.in‚ 619.63671875" disk.nvme0n1.out‚ 60190.6640625 disk.rbd0.in‚0 disk.rbd0.out‚ 3.087890625 disk.nvme1n1.in‚ 498.80078125 disk.sda.out‚ 1271.421875 disk.sdb1.out‚0 disk.sda1.in‚0 disk.md0p1.in‚ 9.26953125 cpu‚25.198384741867535¹:ÿ¦”ò'‚(:‰  ØêÔ⥵ÿ gpu.0.gpu‚56.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚125.603' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚31.40075 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0“1—ÿjó'‚(:ß  ØêÔ§´Ñÿ" disk.sdb2.out‚1273.59130859375 disk.sda1.in‚0 disk.md0p1.out‚ 1227.65234375# disk.nvme0n1.out‚60466.78515625 disk.md0.in‚ 9.26953125$ memory_percent‚92.10419654846191 network.sent‚ 2.4943265e+07 disk.md0p1.in‚ 9.26953125 disk.sda.in‚ 6.08203125" proc.memory.rssMB‚ 3001.5546875 disk.rbd2.in‚0 disk.nvme0n1.in‚ 619.63671875! disk.sdb.out‚1273.59130859375" disk.sda2.out‚1273.59130859375 disk.rbd0.in‚0( proc.memory.availableMB‚ 1293.6484375) disk./.usagePercent‚75.86701232074205 disk.sdb2.in‚3.1875 disk.sdb.in‚3.1875 disk.sdb1.out‚0 cpu‚25.198074474407505 disk.dm-0.in‚ 1118.4375 disk.nvme1n1.in‚ 498.80078125 disk.nvme1n1.out‚61888.25 disk.sdb1.in‚0* proc.memory.percent‚18.320035934448242 network.recv‚ 3.578988e+06$ disk./.usageGB‚2713.371841430664 disk.rbd2.out‚3.25 disk.sda1.out‚0! disk.dm-0.out‚122355.03515625! disk.sda.out‚1273.59130859375 disk.sda2.in‚ 6.08203125 disk.rbd0.out‚ 3.087890625 disk.md0.out‚ 1227.65234375 proc.cpu.threads‚17ûÊ"Ü4ô'/ _wandbruntime‚11141  _runtime‚11141Ù¤Ï{Êõ'Ä ' lr-AdamW/pg1‚0.00010621446025166679 ' lr-AdamW/pg2‚0.00010621446025166679 trainer/global_step‚9599 % _timestamp‚1.7885215159985712e+09 _runtime‚11141.469033065 _step‚785ܹdÏö'É trainer/global_step‚9599 % _timestamp‚1.7885215159985712e+09 _runtime‚11141.469033065 _step‚785 ' lr-AdamW/pg1‚0.00010621446025166679 ' lr-AdamW/pg2‚0.00010621446025166679‘ØY4÷'/ _wandbruntime‚11141  _runtime‚11141Lfwp(ø'€P G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407872 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚788101173757440 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177047040 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 228859904 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26090 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚799002070057472 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11386635320320 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450780 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407872 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚57490713 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 197768172 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _step‚786 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1187 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410816 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955591168 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚628 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚781440241952470 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 240615868 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 180469406 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚57491339 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26088 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10901795707904 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 197768172 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 180470034 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 252800000 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚799002070057472 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10900896300032 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 438357950 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 240615868 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10900896300032 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚893324694744064 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 438384040 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411420 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚770579090366064 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚788101173757440 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10901795707904 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955591168 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚881938035483648 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 122978693 $ _timestamp‚1.788521515999862e+09 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚799002987468288 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26088 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 130379939848Äñ¹ ù'…P QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 197768172 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _step‚786 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1187 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410816 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955591168 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚628 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚781440241952470 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 240615868 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 180469406 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚57491339 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26088 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 [KDeviceStatsMo?ACRnitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10901795707904 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 197768172 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 180470034 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 252800000 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚799002070057472 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10900896300032 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 438357950 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 240615868 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10900896300032 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚893324694744064 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 438384040 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411420 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚770579090366064 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚788101173757440 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10901795707904 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955591168 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚881938035483648 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 122978693 $ _timestamp‚1.788521515999862e+09 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚799002987468288 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26088 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407872 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚788101173757440 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177047040 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 228859904 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26090 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚799002070057472 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11386635320320 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450780 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407872 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚57490713 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712’Ö¹b4ú'/ _wandbruntime‚11142  _runtime‚11142¤\:'û'‹N WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚781519330708124 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚770657053420832 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007241 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10902908524544 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177047040 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10861362836304 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 180488068 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 197788068 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚799083730580480 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11387785722880 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10902009103360 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 438402439 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚893414915774976 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 240640485 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 401960960 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10862260247932 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 240614371 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 122991071 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899421184 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1187 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚788180804052992 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚799082813156352 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚788180804052992 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚882027354704384 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 438428555 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 trainer/global_step‚9599 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10902009103360 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26114 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚621 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚619 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11387561070592 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚781518416257136 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411628 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 240614371 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899421184 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177047040 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7885215169813638e+09 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚788180822055936 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917424128 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 224652288 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚799083730580480 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 122991069 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚57496378 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚893415164367360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 197788070 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955591168 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10902908524544 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26114 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 _runtime‚11142.453073293 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 438402439 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 197788068 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 180487447 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450988 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7885215169813638e+09 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚788180822055936 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917424128 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 224652288 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚799083730580480 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 122991069 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚57496378 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚893415164367360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 197788070 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955591168 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10902908524544 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26114 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 _runtime‚11142.453073293 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 438402439 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 197788068 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 180487447 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450988 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 197788068 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚799083730580480 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11387785722880 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10902009103360 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 438402439 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚893414915774976 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 240640485 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 401960960 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10862260247932 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 240614371 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 122991071 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899421184 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1187 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚788180804052992 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚799082813156352 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚788180804052992 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚882027354704384 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 438428555 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 trainer/global_step‚9599 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10902009103360 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26114 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚621 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚619 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11387561070592 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚781518416257136 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411628“ìiI4ý'/ _wandbruntime‚11142  _runtime‚111421sêr`þ'Ú 0loss/FRt1/assignment_loss‚2.8297547503393523 .loss/FRt2/assignment_loss‚1.52838691284782 2loss/SRqqt2/assignment_loss‚1.0458640806318726 0loss/FBt3/assignment_loss‚0.2111215745812337 /loss/FRt4/detection_loss‚2.9612286526423235 /loss/FBt1/detection_loss‚2.3801067742434414 /loss/FBt4/detection_loss‚2.3240219463001597 %loss/total_loss‚38.64011783032501 1loss/FBt1/assignment_loss‚0.12141224404857356 /loss/FRt2/detection_loss‚1.3929915546455927 2loss/SRqqt1/assignment_loss‚1.3222059269435704 2loss/SRqqt3/assignment_loss‚1.2127319832260792 1loss/SRqqt1/detection_loss‚1.7114496086724102 1loss/SRqqt2/detection_loss‚0.8792175984708592 0loss/SRqqt3/detection_loss‚2.020636445723283 1loss/FBt2/assignment_loss‚0.09373349326989455 1loss/FBt4/assignment_loss‚0.04545061105264639 /loss/FRt3/detection_loss‚2.6144831313229187 /loss/FBt2/detection_loss‚0.6583054276138899 epoch‚7 trainer/global_step‚9599 _step‚788 /loss/FRt1/detection_loss‚2.1075832862914803 0loss/FRt3/assignment_loss‚2.2934506797209018 0loss/FRt4/assignment_loss‚2.4313145004785977 1loss/SRqqt4/detection_loss‚2.1487274914979935 _runtime‚11142.454247329 2loss/SRqqt4/assignment_loss‚1.3191508299205452 /loss/FBt3/detection_loss‚2.9867883258395724 % _timestamp‚1.7885215169840765e+09LÓ‚eÿ'ß /loss/FRt4/detection_loss‚2.9612286526423235 /loss/FBt1/detection_loss‚2.3801067742434414 /loss/FBt4/detection_loss‚2.3240219463001597 %loss/total_loss‚38.64011783032501 1loss/FBt1/assignment_loss‚0.12141224404857356 /loss/FRt2/detection_loss‚1.3929915546455927 2loss/SRqqt1/assignment_loss‚1.3222059269435704 2loss/SRqqt3/assignment_loss‚1.2127319832260792 1loss/SRqqt1/detection_loss‚1.7114496086724102 1loss/SRqqt2/detection_loss‚0.8792175984708592 0loss/SRqqt3/detection_loss‚2.020636445723283 1loss/FBt2/assignment_loss‚0.09373349326989455 1loss/FBt4/assignment_loss‚0.04545061105264639 /loss/FRt3/detection_loss‚2.6144831313229187 /loss/FBt2/detection_loss‚0.6583054276138899 epoch‚7 trainer/global_step‚9599 _step‚788 /loss/FRt1/detection_loss‚2.1075832862914803 0loss/FRt3/assignment_loss‚2.2934506797209018 0loss/FRt4/assignment_loss‚2.4313145004785977 1loss/SRqqt4/detection_loss‚2.1487274914979935 _runtime‚11142.454247329 2loss/SRqqt4/assignment_loss‚1.3191508299205452 /loss/FBt3/detection_loss‚2.9867883258395724 % _timestamp‚1.7885215169840765e+09 0loss/FRt1/assignment_loss‚2.8297547503393523 .loss/FRt2/assignment_loss‚1.52838691284782 2loss/SRqqt2/assignment_loss‚1.0458640806318726 0loss/FBt3/assignment_loss‚0.2111215745812337”ØNÅ”€(‚(:‰ ¯ØêÔ¿Ùÿ gpu.0.gpu‚30.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚36.0 gpu.0.powerWatts‚114.131' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.53275 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¾×‡sm(‚(:â ¯ØêÔŸÏþ€" disk.sda2.out‚1275.37939453125 disk.rbd0.in‚0 disk.nvme1n1.in‚ 498.80078125 disk.sdb2.in‚3.1875" disk.sdb2.out‚1275.37939453125) proc.memory.availableMB‚ 1292.88671875 disk.md0p1.in‚ 9.26953125# disk.nvme1n1.out‚61888.37109375 disk.sdb1.in‚0 disk.md0.in‚ 9.26953125 disk.rbd2.out‚3.25" proc.memory.rssMB‚ 3001.5546875 disk.rbd2.in‚0 disk.sda1.out‚0 disk.nvme0n1.in‚ 619.63671875 disk.md0p1.out‚ 1229.3828125 network.recv‚ 3.589677e+06) disk./.usagePercent‚75.87050033346348 disk.sda.in‚ 6.08203125! disk.sda.out‚1275.37939453125 disk.md0.out‚ 1229.3828125! disk.sdb.out‚1275.37939453125# memory_percent‚92.1088457107544# disk.nvme0n1.out‚60636.15234375 network.sent‚ 2.4952397e+07% disk./.usageGB‚2713.4965896606445 disk.sda2.in‚ 6.08203125 disk.sdb1.out‚0 disk.dm-0.out‚122524.5234375* proc.memory.percent‚18.320035934448242 cpu‚25.19782444836898 disk.dm-0.in‚ 1118.4375 disk.sdb.in‚3.1875 disk.rbd0.out‚ 3.087890625 disk.sda1.in‚0 proc.cpu.threads‚17‹f”‚(‚(:‰ ¾ØêÔöõ´ÿ gpu.0.gpu‚38.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚159.079' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚39.76975 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0f_ ·xƒ(‚(:Ú ¾ØêÔÌ“à€ disk.sda1.in‚0 disk.sdb1.out‚0" proc.memory.rssMB‚ 3001.5546875 cpu‚25.19753796480551 network.sent‚ 2.506966e+07 disk.dm-0.in‚ 1118.4375! disk.nvme0n1.out‚ 60672.203125 disk.sdb.in‚3.1875) disk./.usagePercent‚75.87398813286387 disk.rbd2.in‚0! disk.sdb2.out‚1276.7919921875! disk.sda2.out‚1276.7919921875 disk.nvme1n1.in‚ 498.80078125 proc.cpu.threads‚17 disk.md0p1.out‚ 1230.76171875 disk.sdb2.in‚3.1875 disk.md0p1.in‚ 9.26953125 disk.sdb.out‚1276.7919921875 disk.sdb1.in‚0 network.recv‚ 3.609157e+06% disk./.usageGB‚ÂxóJí2713.6213302612305 disk.sda1.out‚0! disk.dm-0.out‚122560.75390625 disk.rbd0.in‚0 disk.rbd0.out‚ 3.087890625 disk.sda.out‚1276.7919921875 disk.sda2.in‚ 6.08203125# disk.nvme1n1.out‚61888.55078125$ memory_percent‚92.11196899414062* proc.memory.percent‚18.320035934448242 disk.md0.in‚ 9.26953125 disk.md0.out‚ 1230.76171875 disk.nvme0n1.in‚ 619.63671875 disk.sda.in‚ 6.08203125$ proc.memory.availableMB‚1292.375 disk.rbd2.out‚3.25Ú=ª”„(‚(:‰ ÍØêÔÚ¦ÿ gpu.0.gpu‚44.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚37.0 gpu.0.powerWatts‚164.989' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚41.24725 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0?¹þj…(‚(:ß ÍØêÔޤˀ disk.dm-0.in‚ 1118.4375 disk.sdb1.out‚0# disk.nvme1n1.out‚61888.73828125 network.recv‚ 3.612501e+06) disk./.usagePercent‚75.87400178541198 disk.sda2.in‚ 6.08203125) proc.memory.availableMB‚ 1293.08203125* proc.memory.percent‚18.320035934448242 disk.rbd0.out‚ 3.087890625 disk.sdb1.in‚0" proc.memory.rssMB‚ 3001.5546875" disk.nvme0n1.out‚ 61199.5078125 disk.md0.out‚ 1232.8046875 disk.rbd2.out‚3.25 cpu‚25.19727417683377 network.sent‚ 2.5075017e+07! disk.sdb2.out‚1278.8779296875 disk.sda1.in‚0! disk.dm-0.out‚123088.24609375 disk.rbd2.in‚0% disk./.usageGB‚2713.6218185424805 disk.sdb2.in‚3.1875 disk.sda1.out‚0 disk.md0p1.out‚ 1232.8046875 disk.nvme0n1.in‚ 619.63671875 disk.sda.out‚1278.8779296875 disk.rbd0.in‚0 disk.nvme1n1.in‚ 498.80078125 disk.md0p1.in‚ 9.26953125 disk.sdb.in‚3.1875 disk.sdb.out‚1278.8779296875! disk.sda2.out‚1278.8779296875$ memory_percent‚92.10765361785889 proc.cpu.threads‚17 disk.sda.in‚ 6.08203125 disk.md0.in‚ 9.26953125ì °„”†(‚(:‰ ÜØêÔ¢¤Éÿ gpu.0.gpu‚20.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚164.039' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚41.00975 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0>š»j‡(‚(:ß ÜØêÔäýÄ€ disk.sda.in‚ 6.08203125$ memory_percent‚92.10708141326904" proc.memory.rssMB‚ 3001.5546875 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.rbd2.out‚3.25 disk.sdb.in‚3.1875 disk.sdb2.in‚3.1875 disk.md0p1.in‚ 9.26953125 disk.rbd0.in‚0 disk.rbd2.in‚0! disk.sdb2.out‚1280.4384765625 disk.sda.out‚1280.4384765625 disk.nvme1n1.in‚ 498.80078125 disk.dm-0.in‚ 1118.4375 cpu‚25.196988290332275 disk.md0.in‚ 9.26953125 disk.nvme0n1.in‚ 619.63671875 disk.rbd0.out‚ 3.087890625) proc.memory.availableMB‚ 1293.17578125* proc.memory.percent‚18.320035934448242 network.recv‚ 3.617064e+06 disk.md0.out‚ 1234.31640625! disk.dm-0.out‚123425.35546875 disk.sdb.out‚1280.4384765625 disk.sda2.in‚ 6.08203125! disk.sda2.out‚1280.4384765625 network.sent‚ 2.5078792e+07) disk./.usagePercent‚75.88099189004511# disk.nvme1n1.out‚61888.91796875 disk.sda1.out‚0 disk.md0p1.out‚ 1234.31640625 disk.nvme0n1.out‚ 61536.4375 proc.cpu.threads‚17% disk./.usageGB‚2713.8718185424805 disk.sda1.in‚0:wݸ4ˆ(/ _wandbruntime‚11200  _runtime‚11200°¿Pwʉ(Ä ' lr-AdamW/pg2‚0.00010216969298826262 trainer/global_step‚9649 % _timestamp‚1.7885215749993482e+09 _runtime‚11200.469873259 _step‚789 ' lr-AdamW/pg1‚0.00010216969298826262X—=åÏŠ(É ' lr-AdamW/pg1‚0.00010216969298826262 ' lr-AdamW/pg2‚0.00010216969298826262 trainer/global_step‚9649 % _timestamp‚1.7885215749993482e+09 _runtime‚11200.469873259 _step‚789•Îeï=4‹(/ _wandbruntime‚11200  _runtime‚11200Ž©(Œ(€P UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚803039239415296 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 181375229 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 123597495 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 440585752 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26088 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚626 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚774478078033104 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚803040156826112 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚57777736 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 181375857 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 198763070 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚897848740545024 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450780 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007241 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1212 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177047040 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955591168 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚57778362 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411420 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 198763070 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 198763072 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚785394693315978 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚803039239415296 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚792082706537984 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10956550880256 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26088 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚792082688535040 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚792082706537984 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚628 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 123597493 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11444744024576 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917410816 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 241848770 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410816 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11444972884480 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10957450288128 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10916598243514 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 trainer/global_step‚9649 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407872 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚792082688535040 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 241848770 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0¦ ìN ((…P GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 181375857 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 198763070 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚897848740545024 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450780 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007241 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1212 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177047040 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955591168 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚57778362 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411420 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 198763070 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 198763072 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚785394693315978 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚803039239415296 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚792082706537984 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10956550880256 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26088 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚792082688535040 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚792082706537984 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚628 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 123597493 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11444744024576 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917410816 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 241848770 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410816 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11444972884480 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10957450288128 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10916598243514 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 trainer/global_step‚9649 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407872 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚792082688535040 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 241848770 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚803039239415296 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 181375229 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 123597495 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 440585752 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26088 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚626 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚774478078033104 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚803040156826112 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚57777736 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564–›ç4Ž(/ _wandbruntime‚11201  _runtime‚11201v±§Yv(‹N ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚774556075166592 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 % _timestamp‚1.7885215759542964e+09 _runtime‚11201.425432507 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11445922947584 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 241847273 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚610 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚886493339681280 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 241873387 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10917706905040 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31686 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚57784058 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 241873387 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 123609869 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚897939474698240 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚608 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚774556058127232 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 440630241 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 198782968 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 440630241 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚785473782071632 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411628 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899421184 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 440656357 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 123609871 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450988 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 181393319 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚792162318830592 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899421184 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚886493315741184 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚9649 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 181393929 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚785472867620644 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 241847273 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10958563104768 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 198782970 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10958563104768 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 440656357 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177047040 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚57783450 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007241 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917424128 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚Š-›607 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 198782970 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26114 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚803120899938304 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚803120899938304 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 401960960 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31686 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚57784058 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 241873387 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 123609869 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚897939474698240 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚608 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚774556058127232 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 440630241 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 198782968 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 440630241 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚785473782071632 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411628 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899421184 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 440656357 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 123609871 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450988 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 181393319 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚792162318830592 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899421184 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚886493315741184 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚9649 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 181393929 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚785472867620644 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 241847273 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10958563104768 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 198782970 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10958563104768 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 440656357 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177047040 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚57783450 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007241 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917424128 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 198782970 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26114 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚803120899938304 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚803120899938304 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 401960960 BDeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 D>DeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 MDDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.peak‚1212 eQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.allocated‚778710371638784 ]NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.current‚ 1182793728 ZFDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.freed‚807545505420800 SEDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.freed‚ 182520622 LDDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.peak‚564 aMDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.freed‚796539177845760 VHDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.freed‚ 199969248 GADeviceStatsMonitor.on_validation_batch_start/num_sync_all_streams‚2 VHDeviceStatsMonitor.on_validation_batch_start/active.large_pool.allocated‚ 199969250 RDDeviceStatsMonitor.on_validation_batch_start/active.large_pool.freed‚ 199969248 VNDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.current‚199 E?DeviceStatsMonitor.on_validation_batch_start/num_oom_rejections‚0 NHDeviceStatsMonitor.on_validation_batch_start/oversize_se)+M¼¨gments.allocated‚0 JDDeviceStatsMonitor.on_validation_batch_start/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.freed‚807545505420800 \LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 J@DeviceStatsMonitor.on_validation_batch_start/allocation.all.peak‚31931 PGDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.peak‚3677 OGDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.current‚201 QKDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.allocated‚0 I@DeviceStatsMonitor.on_validation_batch_start/segment.all.current‚1171 trainer/global_step‚9671 I?DeviceStatsMonitor.on_validation_batch_start/active.all.current‚20521 RDDeviceStatsMonitor.on_validation_batch_start/active.small_pool.freed‚ 242823908 VHDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.freed‚ 242823908 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.allocated‚58176120 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 UEDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.peak‚ 13955591168 LFDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.peak‚0 UEDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.peak‚ 13510214665 \ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.current‚17039360 MCDeviceStatsMonitor.on_validation_batch_start/allocation.all.current‚20521 dQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.allocated‚10965976407380 [KDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 RBDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.peak‚ 13955591168 ^JDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.freed‚796539177845760 ^JDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.allocated‚807546198540288 \LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.peak‚ 12392107008 OGDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.current‚607.Zs*¢(íT TKDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.peak‚1187 XJDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.peak‚ 401960960 aMDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.freed‚778710354599424 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.freed‚0 _PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.freed‚58175921 VHDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.current‚ 690184120 FDeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 D>DeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 MDDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.peak‚1212 eQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.allocated‚778710371638784 ]NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.current‚ 1182793728 ZFDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.freed‚807545505420800 SEDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.freed‚ 182520622 LDDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.peak‚564 aMDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.freed‚796539177845760 VHDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.freed‚ 199969248 GADeviceStatsMonitor.on_validation_batch_start/num_sync_all_streams‚2 VHDeviceStatsMonitor.on_validation_batch_start/active.large_pool.allocated‚ 199969250 RDDeviceStatsMonitor.on_validation_batch_start/active.large_pool.freed‚ 199969248 VNDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.current‚199 E?DeviceStatsMonitor.on_validation_batch_start/num_oom_rejections‚0 NHDeviceStatsMonitor.on_validation_batch_start/oversize_segments.allocated‚0 JDDeviceStatsMonitor.on_validation_batch_start/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.freed‚807545505420800 \LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 J@DeviceStatsMonitor.on_validation_batch_start/allocation.all.peak‚31931 PGDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.peak‚3677 OGDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.current‚201 QKDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.allocated‚0 I@DeviceStatsMonitor.on_validation_batch_start/segment.all.current‚1171 trainer/global_step‚9671 I?DeviceStatsMonitor.on_validation_batch_start/active.all.current‚20521 RDDeviceStatsMonitor.on_validation_batch_start/active.small_pool.freed‚ 242823908 VHDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.freed‚ 242823908 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.allocated‚58176120 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 UEDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.peak‚ 13955591168 LFDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.peak‚0 UEDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.peak‚ 13510214665 \ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.current‚17039360 MCDeviceStatsMonitor.on_validation_batch_start/allocation.all.current‚20521 dQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.allocated‚10965976407380 [KDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 RBDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.peak‚ 13955591168 ^JDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.freed‚796539177845760 ^JDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.allocated‚807546198540288 \LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.peak‚ 12392107008 OGDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.current‚607 \ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.current‚18002944 `MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.freed‚11006327575040 TJDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.current‚20519 fRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.freed‚891175362725376 F=DeviceStatsMonitor.on_validation_batch_start/num_device_alloc‚1171 ^NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.current‚ 13037993984 MCDeviceStatsMonitor.on_validation_batch_start/active.small_pool.peak‚31686 ]JDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.freed‚11006327575040 _KDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.freed‚902671077989888 ]ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.current‚ 673144760 _runtime‚11281.323967152 aNDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.allocated‚11007002691584 ZLDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.allocated‚ 242844427 jVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.allocated‚891175386665472 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 KEDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.freed‚0 ^PDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.allocated‚ 124344703 [MDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.current‚ 170907136 ^JDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.allocated‚789676348046164 WGDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.current‚ 14220787712 B;DeviceStatsMonitor.on_validation_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_validation_batch_start/oversize_segments.peak‚0 ZFDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.freed‚789675657862044 ZKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 RKDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.peak‚35 VHDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.current‚ 693119488 eRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.freed‚11495715264512 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.freed‚0 ZLDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.allocated‚ 199969250 QGDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.peak‚31686 OIDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.current‚0 `MDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.freed‚10965303262620™¼)Š4£(/ _wandbruntime‚11282  _runtime‚11282ª¹`w)¤(ñR VHDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.peak‚ 401960960 93DeviceStatsMonitor.on_validation_batch_end/num_ooms‚0 cODeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.allocated‚778760141728608 JBDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.peak‚564 _step‚794 G=DeviceStatsMonitor.on_validation_batch_end/active.all.current‚20521 KADeviceStatsMonitor.on_validation_batch_end/active.small_pool.peak‚31686 ZJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 gTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.allocated‚11496350794240 M?DeviceStatsMonitor.on_validation_batch_end/active.all.allocated‚ 442837727 YJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.peak‚ 1175007241 ^NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 $ _timestamp‚1.788521657031723e+09 PBDeviceStatsMonitor.on_validation_batch_end/active.large_pool.freed‚ 199984524 bODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.allocated‚11007491254272 I@DeviceStatsMonitor.on_validation_batch_end/segment.all.allocated‚1171 D;DeviceStatsMonitor.on_validation_batch_end/segment.all.peak‚1171 trainer/global_step‚9671 NDDeviceStatsMonitor.on_validation_batch_end/active.small_pool.current‚20519 YJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.peak‚ 1177047040 RHDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.current‚20519 PIDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.peak‚35 OGDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.allocated‚564 hTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.allocated‚891230836137472 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 YIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 XIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 TFDeviceStatsMonitor.on_validation_batch_end/active.small_pool.allocated‚ 242853201 \HDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.freed‚796592749121536 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.freed‚ 182535613 @9DeviceStatsMonitor.on_validation_batch_end/max_split_size‚-1 RBDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.peak‚ 14220787712 JADeviceStatsMonitor.on_validation_batch_end/active.large_pool.peak‚3677 ^KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.freed‚10965790189344 JDDeviceStatsMonitor.on_validation_batch_end/active.large_pool.current‚2 OEDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.peak‚31686 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 SCDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.peak‚ 13510214665 XDDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.freed‚807599565259264 RIDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.peak‚1187 OIDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.current‚18002944 TFDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.freed‚ 199984524 ]NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 `LDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.allocated‚796592767124480 _RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.current‚23940096 JDDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.peak‚0 XJDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.allocated‚ 199984526 \HDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.allocated‚789726605062712 BDeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚31931 XJDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.freed‚ 124354141 C=DeviceStatsMonitor.on_validation_batch_end/num_oom_rejections‚0 YEDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.allocated‚807600258378752 _LDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.allocated‚11007491254272 aMDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.allocated‚902727186931712 WGDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.allocated‚ 14220787712 MEDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.current‚564 ^KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.freed‚11006816137728 MGDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.freed‚0 G>DeviceStatsMonitor.on_validation_batch_end/segment.all.current‚1171 OGDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.allocated‚607 PBDeviceStatsMonitor.on_validation_batch_end/active.small_pool.freed‚ 242832682 [HDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.freed‚11006816137728 ZMDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.current‚18002944 _KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.freed‚796592749121536 [MDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.current‚ 675116544 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.allocated‚58181671 BDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚1171 OGDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.allocated‚607 PBDeviceStatsMonitor.on_validation_batch_end/active.small_pool.freed‚ 242832682 [HDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.freed‚11006816137728 ZMDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.current‚18002944 _KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.freed‚796592749121536 [MDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.current‚ 675116544 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.allocated‚58181671 BDeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚31931 XJDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.freed‚ 124354141 C=DeviceStatsMonitor.on_validation_batch_end/num_oom_rejections‚0 YEDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.allocated‚807600258378752 _LDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.allocated‚11007491254272 aMDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.allocated‚902727186931712 WGDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.allocated‚ 14220787712 MEDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.current‚564š'®†³’¦(‚(:‡ ÅÙêÔ×Ŧÿ gpu.0.gpu‚2.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚35.0 gpu.0.powerWatts‚113.466' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.3665 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0j¥º¯m§(‚(:â ÅÙêÔ»¡‚ disk.md0p1.in‚ 9.26953125# disk.nvme0n1.out‚62450.67578125# disk.nvme1n1.out‚62034.64453125 disk.rbd2.out‚3.25* proc.memory.percent‚18.320035934448242! disk.sdb.out‚1292.25146484375 disk.nvme1n1.in‚ 498.80078125 disk.sdb1.out‚0 disk.md0.out‚ 1245.703125) proc.memory.availableMB‚ 1293.05859375" proc.memory.rssMB‚ 3001.5546875 network.sent‚ 2.5237027e+07 disk.sdb2.in‚3.1875 disk.sda1.out‚0) disk./.usagePercent‚75.83898417277163 disk.nvme0n1.in‚ 619.63671875" disk.sda2.out‚1292.25146484375 disk.rbd0.in‚0$ memory_percent‚92.10779666900635! disk.sda.out‚1292.25146484375 disk.sda1.in‚0% disk./.usageGB‚2712.3694190979004 disk.md0p1.out‚ 1245.703125 disk.sda.in‚ 6.08203125" disk.sdb2.out‚1292.25146484375 disk.dm-0.in‚ 1118.4375 proc.cpu.threads‚17 disk.sdb1.in‚0 disk.rbd2.in‚0 disk.dm-0.out‚124449.0703125 cpu‚25.220638701788378 network.recv‚ 3.653734e+06 disk.sdb.in‚3.1875 disk.sda2.in‚ 6.08203125 disk.rbd0.out‚ 3.087890625 disk.md0.in‚ 9.26953125 µÇ4¨(/ _wandbruntime‚11298  _runtime‚11298˜ºþEs©(¦ ;"Purity/*FRt3SRqqt*FBt/event_purity‚0.008064515888690948 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.2752251923084259 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.24897120893001556 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 particle/accuracy_4_of_7‚1 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9922114014625549 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9924264550209045 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.22198931872844696 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9938936829566956 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.2718910574913025 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 particle/accuracy_2_of_5‚1 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 ,"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.375 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 ;"Purity/0FRt3SRqqt*FBt/event_purity‚0.026143791154026985 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.4211050271987915 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.6060606241226196 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3996489346027374 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5462067127227783 -"Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.3125 jet/accuracy_6_of_6‚0 & Purity/*FRt3SRqqt2FBt/FRt_purity‚1 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9919393062591553 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.5163926482200623 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 & Purity/0FRt3SRqqt*FBt/FBt_purity‚1 + Purity/3FRt3SRqqt*FBt/FRt_purity‚0.3125 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 *jet/accuracy_1_of_5‚0.9536802768707275 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 6 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.193452388048172 8"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.392642617225647 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 ("Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9974541664123535 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9919427633285522 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 (particle/f_score‚0.43936750292778015 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 6 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.248198002576828 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.23641757667064667 6 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.983730137348175 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.04032101109623909 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 jet/accuracy_7_of_7‚0 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 particle/accuracy_2_of_2‚1 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.02926219440996647 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 :"Purity/2FRt0SRqqt2FBt/event_purity‚0.02631578966975212 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 *jet/accuracy_2_of_3‚0.2439962923526764 9"Purity/0FRt0SRqqt3FBt/event_purity‚0.1197916641831398 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.3974530100822449 9"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.4645061790943146 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.17442528903484344 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.22699232399463654 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.4891025424003601 7 Purity/1FRt0SRqqt1FBt/FBt_purity‚0.9988662600517273 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 ,"Purity/1FRt0SRqqt3FBt/event_purity‚0.125 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.26691657304763794 (particle/accuracy‚0.8570032715797424 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.20000000298023224 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 :"Purity/0FRt*SRqqt0FBt/event_purity‚0.31771039962768555 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.05446028336882591 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 particle/accuracy_3_of_6‚1 8 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.48355263471603394 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 particle/accuracy_7_of_7‚1 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.2837035655975342 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 :"Purity/0FRt0SRqqt2FBt/event_purity‚0.35850104689598083 *jet/accuracy_1_of_4‚0.9153264760971069 *jet/accuracy_3_of_4‚0.0688389241695404 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9985569715499878 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 7 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.1428571492433548 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 8 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.42705875635147095 8 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.44663581252098083 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 ) Purity/4FRt0SRqqt*FBt/FRt_purity‚0.25 :"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.43130749464035034 :"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.41940590739250183 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.6070636510848999 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.02870428003370762 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.30863603949546814 7 Purity/1FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.009826152585446835 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.22212643921375275 * Purity/*FRt*SRqqt4FBt/FBt_purity‚0.875 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 particle/accuracy_1_of_7‚1 +"Purity/1FRt1SRqqt3FBt/SRqqt_purity‚0.75 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6768600344657898 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.4367649555206299 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 *"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.5 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9977324604988098 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.28964436054229736 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.2708359658718109 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.6666666865348816 jet/accuracy_1_of_6‚1 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 ;"Purity/1FRt1SRqqt1FBt/event_purity‚0.028721753507852554 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.3333333432674408 jet/accuracy_2_of_7‚0.5 7 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.9047619104385376 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.2261768877506256 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.4907407760620117 particle/accuracy_6_of_6‚1 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.4167962074279785 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 :"Purity/*FRt0SRqqt2FBt/event_purity‚0.25367599725723267 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.23851284384727478 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.1797594428062439 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.5308400392532349 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.10561291128396988 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.011136923916637897 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 particle/accuracy_1_of_4‚1 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 *"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.5 _step‚795 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9951229691505432 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 ("Purity/*FRt2SRqqt3FBt/SRqqt_purity‚1 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.3340700566768646 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.3934588134288788 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 7 Purity/3FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 :"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.48922693729400635 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 particle/accuracy_2_of_3‚1 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.10087718814611435 *"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.5 7 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.2666666805744171 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 :"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.39697855710983276 *jet/accuracy_1_of_2‚0.7366651892662048 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.4345453381538391 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 8"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.449126273393631 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9941520690917969 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.3611607253551483 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 8 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.29629629850387573 +"Purity/1FRt*SRqqt3FBt/SRqqt_purity‚0.75 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9897977113723755 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 ;"Purity/0FRt3SRqqt0FBt/event_purity‚0.026143791154026985 particle/accuracy_5_of_7‚1 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 & Purity/*FRt3SRqqt*FBt/FBt_purity‚1 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 jet/accuracy_1_of_7‚1 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.025416988879442215 :"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.42831137776374817 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.2905193269252777 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 *"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.5 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 :"Purity/*FRt*SRqqt1FBt/event_purity‚0.40240541100502014 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 jet/accuracy_5_of_5‚0 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.5714285969734192 8 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.44736841320991516 7 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.3529411852359772 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.018291493877768517 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.014624316245317459 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.4292447865009308 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 +jet/accuracy_3_of_5‚0.08961213380098343 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 & Purity/*FRt*SRqqt4FBt/FRt_purity‚0 & Purity/*FRt0SRqqt4FBt/FRt_purity‚0 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 * Purity/0FRt0SRqqt4FBt/FBt_purity‚0.875 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9896175265312195 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9826388955116272 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 jet/accuracy_5_of_7‚0 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9351851940155029 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.21281494200229645 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.6716269254684448 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 particle/accuracy_4_of_4‚1 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.3541666567325592 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 particle/accuracy_3_of_7‚1 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.4444444477558136 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.1768796443939209 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 ) Purity/3FRt1SRqqt2FBt/FRt_purity‚0.25 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.1870642751455307 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6443641185760498 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.3988667130470276 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 7 Purity/3FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.12411347776651382 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.06808905303478241 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.22727273404598236 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.3333333432674408 & Purity/0FRt3SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9267140030860901 6 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.932539701461792 7 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.2293630987405777 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.4422784745693207 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.4659722149372101 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.4577394127845764 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 *jet/accuracy_2_of_4‚0.3795573115348816 , Purity/1FRt3SRqqt1FBt/FRt_purity‚0.21875 :"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.47333332896232605 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 & Purity/0FRt2SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9930968284606934 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.4525619149208069 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9983749389648438 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.06104941666126251 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.5757575631141663 *"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.5 & Purity/1FRt*SRqqt4FBt/FRt_purity‚0 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.4836341142654419 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9942528009414673 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.6111111044883728 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 ,"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.375 ("Purity/4FRt*SRqqt0FBt/SRqqt_purity‚1 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.10393895208835602 +jet/accuracy_1_of_1‚0.44874340295791626 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.23435352742671967 :"Purity/0FRt1SRqqt2FBt/event_purity‚0.02545155957341194 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.3317834436893463 <"Purity/1FRt2SRqqt*FBt/event_purity‚0.0010582010727375746 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.26539796590805054 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.21494710445404053 +jet/accuracy_4_of_6‚0.03846153989434242 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.16631300747394562 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9923393726348877 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9931841492652893 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.4985632300376892 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 % _timestamp‚1.7885216730399127e+09 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.4413106441497803 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 8 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.31018513441085815 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 9"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.3499999940395355 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.2698027491569519 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.12634176015853882 ) Purity/*FRt2SRqqt2FBt/FRt_purity‚0.25 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.3916613459587097 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.20000000298023224 & Purity/1FRt3SRqqt2FBt/FRt_purity‚1 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.3291351795196533 :"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.43125003576278687 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 9"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.4057988226413727 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.2848152816295624 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 7 Purity/1FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 ,jet/accuracy_3_of_3‚0.017733128741383553 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.7857142686843872 ("Purity/4FRt*SRqqt*FBt/SRqqt_purity‚1 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 particle/accuracy_1_of_1‚1 ) Purity/2FRt3SRqqt1FBt/FRt_purity‚0.25 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.21773815155029297 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.2288377285003662 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 ) Purity/4FRt0SRqqt0FBt/FRt_purity‚0.25 trainer/global_step‚9671 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.4311671555042267 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.2261768877506256 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.26425546407699585 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 particle/accuracy_4_of_6‚1 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 particle/accuracy_3_of_3‚1 particle/accuracy_2_of_4‚1 particle/accuracy_5_of_6‚1 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 +jet/accuracy_2_of_2‚0.10695594549179077 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.4374999701976776 ("Purity/0FRt2SRqqt3FBt/SRqqt_purity‚1 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 -jet/accuracy_4_of_4‚0.0011337868636474013 +particle/specificity‚0.9491193294525146 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.22478629648685455 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.9210526347160339 particle/accuracy_6_of_7‚1 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.2081618458032608 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.48656174540519714 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9969576597213745 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚1 9"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.4023568332195282 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9985979795455933 6 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.999546468257904 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.10147566348314285 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.1388888955116272 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 8 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.42716342210769653 particle/accuracy_1_of_3‚1 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9375000596046448 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 :"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.39517107605934143 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9922183752059937 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 jet/accuracy_6_of_7‚0 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 :"Purity/0FRt*SRqqt2FBt/event_purity‚0.27942755818367004 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.2411462962627411 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.34355494379997253 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.4116036593914032 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.3541666567325592 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9892512559890747 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 8"Purity/2FRt*SRqqt1FBt/event_purity‚0.027286471799016 jet/accuracy_5_of_6‚0 particle/accuracy_2_of_6‚1 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.2857142984867096 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.5690003037452698 8 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.45645979046821594 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.31041663885116577 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.3887869417667389 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 + Purity/3FRt3SRqqt0FBt/FRt_purity‚0.3125 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9896155595779419 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 *jet/accuracy_1_of_3‚0.8684682846069336 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 & Purity/*FRt3SRqqt1FBt/FBt_purity‚1 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 7 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.9166666865348816 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 ;"Purity/*FRt3SRqqt0FBt/event_purity‚0.008064515888690948 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5696281790733337 7 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5227272510528564 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.2778644859790802 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2972036302089691 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 +particle/sensitivity‚0.3591760993003845 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 *"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.5 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.3615521192550659 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.22854205965995789 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.4930046796798706 9"Purity/0FRt2SRqqt1FBt/event_purity‚0.0194968543946743 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.3143133521080017 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.0610119067132473 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.4062500298023224 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 <"Purity/1FRt2SRqqt0FBt/event_purity‚0.0012210012646391988 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.3916613459587097 7 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.1716374307870865 particle/accuracy_3_of_5‚1 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 -"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.5625 :"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.39912864565849304 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.4285714328289032 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 particle/accuracy_3_of_4‚1 7 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.2608024477958679 jet/accuracy_4_of_7‚0 =&Purity/*FRt1SRqqt0FBt/event_proportRUvoÃion‚0.3304191529750824 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.3926376700401306 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.2801065146923065 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.1944444626569748 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8209079504013062 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 ) Purity/*FRt1SRqqt3FBt/FRt_purity‚0.75 8"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.408194363117218 :"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.40834441781044006 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.4858374297618866 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9913443922996521 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 +jet/accuracy_3_of_6‚0.11538461595773697 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 8 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.49418604373931885 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.5416666865348816 ) Purity/4FRt1SRqqt0FBt/FRt_purity‚0.25 ;"Purity/*FRt1SRqqt2FBt/event_purity‚0.013076036237180233 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5329343676567078 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.30010291934013367 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.1111111119389534 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.07822013646364212 ;"Purity/2FRt*SRqqt2FBt/event_purity‚0.023255813866853714 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 particle/accuracy_4_of_5‚1 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 :"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.41153234243392944 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.24633122980594635 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.6666666865348816 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9900730848312378 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 :"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.49000000953674316 + Purity/4FRt*SRqqt*FBt/FRt_purity‚0.1875 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.34375 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.5178161263465881 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 particle/accuracy_1_of_6‚1 ) Purity/1FRt1SRqqt3FBt/FRt_purity‚0.75 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.49058496952056885 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.1930275410413742 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.22331736981868744 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 & Purity/1FRt0SRqqt4FBt/FRt_purity‚0 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.3541666865348816 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 *jet/accuracy_2_of_5‚0.5417946577072144 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 jet/accuracy_3_of_7‚0 ,"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.625 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚1 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.016231132671236992 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9877450466156006 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9898438453674316 +jet/accuracy_2_of_6‚0.46794870495796204 *"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.5 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.27196967601776123 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.23754791915416718 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 *loss/val_total_loss‚38.092942666424854 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5012329816818237 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.22888891398906708 & Purity/2FRt2SRqqt2FBt/FRt_purity‚0 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.1944444626569748 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9746543765068054 ) Purity/4FRt1SRqqt*FBt/FRt_purity‚0.25 + Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9375 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_1_of_5‚1 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 particle/accuracy_1_of_2‚1 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 epoch‚7 :"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.47988155484199524 9"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.4292493760585785 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 8 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.23333331942558289 9"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.4320988059043884 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.2777777910232544 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5311185717582703 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.5416666865348816 + Purity/4FRt*SRqqt0FBt/FRt_purity‚0.1875 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 particle/accuracy_2_of_7‚1 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.5830356478691101 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 ,jet/accuracy_4_of_5‚0.005824372638016939 6validation_average_jet_accuracy‚0.4178694784641266 9"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.3867102563381195 6 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.949999988079071 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.25928962230682373 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 particle/accuracy_5_of_5‚1 9"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.6499999761581421 _runtime‚ 11298.5130676 :"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.40795016288757324 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 9"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.3499999940395355 *"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.5 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.9259259700775146ÙRo&/`ª(†¦ :"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.41153234243392944 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.24633122980594635 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.6666666865348816 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9900730848312378 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 :"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.49000000953674316 + Purity/4FRt*SRqqt*FBt/FRt_purity‚0.1875 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.34375 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.5178161263465881 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 particle/accuracy_1_of_6‚1 ) Purity/1FRt1SRqqt3FBt/FRt_purity‚0.75 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.49058496952056885 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.1930275410413742 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.22331736981868744 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 & Purity/1FRt0SRqqt4FBt/FRt_purity‚0 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.3541666865348816 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 *jet/accuracy_2_of_5‚0.5417946577072144 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 jet/accuracy_3_of_7‚0 ,"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.625 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚1 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.016231132671236992 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9877450466156006 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9898438453674316 +jet/accuracy_2_of_6‚0.46794870495796204 *"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.5 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.27196967601776123 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.23754791915416718 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 *loss/val_total_loss‚38.092942666424854 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5012329816818237 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.22888891398906708 & Purity/2FRt2SRqqt2FBt/FRt_purity‚0 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.1944444626569748 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9746543765068054 ) Purity/4FRt1SRqqt*FBt/FRt_purity‚0.25 + Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9375 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_1_of_5‚1 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 particle/accuracy_1_of_2‚1 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 epoch‚7 :"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.47988155484199524 9"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.4292493760585785 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 8 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.23333331942558289 9"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.4320988059043884 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.2777777910232544 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5311185717582703 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.5416666865348816 + Purity/4FRt*SRqqt0FBt/FRt_purity‚0.1875 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 particle/accuracy_2_of_7‚1 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.5830356478691101 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 ,jet/accuracy_4_of_5‚0.005824372638016939 6validation_average_jet_accuracy‚0.4178694784641266 9"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.3867102563381195 6 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.949999988079071 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.25928962230682373 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 particle/accuracy_5_of_5‚1 9"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.6499999761581421 _runtime‚ 11298.5130676 :"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.40795016288757324 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 9"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.3499999940395355 *"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.5 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.9259259700775146 ;"Purity/*FRt3SRqqt*FBt/event_purity‚0.008064515888690948 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.2752251923084259 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.24897120893001556 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 particle/accuracy_4_of_7‚1 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9922114014625549 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9924264550209045 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.22198931872844696 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9938936829566956 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.2718910574913025 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 particle/accuracy_2_of_5‚1 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 ,"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.375 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 ;"Purity/0FRt3SRqqt*FBt/event_purity‚0.026143791154026985 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.4211050271987915 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.6060606241226196 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3996489346027374 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5462067127227783 -"Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.3125 jet/accuracy_6_of_6‚0 & Purity/*FRt3SRqqt2FBt/FRt_purity‚1 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9919393062591553 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.5163926482200623 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 & Purity/0FRt3SRqqt*FBt/FBt_purity‚1 + Purity/3FRt3SRqqt*FBt/FRt_purity‚0.3125 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 *jet/accuracy_1_of_5‚0.9536802768707275 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 6 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.193452388048172 8"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.392642617225647 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 ("Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9974541664123535 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9919427633285522 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 (particle/f_score‚0.43936750292778015 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 6 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.248198002576828 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.23641757667064667 6 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.983730137348175 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.04032101109623909 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 jet/accuracy_7_of_7‚0 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 particle/accuracy_2_of_2‚1 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.02926219440996647 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 :"Purity/2FRt0SRqqt2FBt/event_purity‚0.02631578966975212 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 *jet/accuracy_2_of_3‚0.2439962923526764 9"Purity/0FRt0SRqqt3FBt/event_purity‚0.1197916641831398 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.3974530100822449 9"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.4645061790943146 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.17442528903484344 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.22699232399463654 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.4891025424003601 7 Purity/1FRt0SRqqt1FBt/FBt_purity‚0.9988662600517273 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 ,"Purity/1FRt0SRqqt3FBt/event_purity‚0.125 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.26691657304763794 (particle/accuracy‚0.8570032715797424 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.20000000298023224 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 :"Purity/0FRt*SRqqt0FBt/event_purity‚0.31771039962768555 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.05446028336882591 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 particle/accuracy_3_of_6‚1 8 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.48355263471603394 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 particle/accuracy_7_of_7‚1 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.2837035655975342 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 :"Purity/0FRt0SRqqt2FBt/event_purity‚0.35850104689598083 *jet/accuracy_1_of_4‚0.9153264760971069 *jet/accuracy_3_of_4‚0.0688389241695404 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9985569715499878 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 7 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.1428571492433548 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 8 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.42705875635147095 8 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.44663581252098083 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 ) Purity/4FRt0SRqqt*FBt/FRt_purity‚0.25 :"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.43130749464035034 :"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.41940590739250183 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.6070636510848999 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.02870428003370762 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.30863603949546814 7 Purity/1FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.009826152585446835 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.22212643921375275 * Purity/*FRt*SRqqt4FBt/FBt_purity‚0.875 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 particle/accuracy_1_of_7‚1 +"Purity/1FRt1SRqqt3FBt/SRqqt_purity‚0.75 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6768600344657898 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.4367649555206299 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 *"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.5 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9977324604988098 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.28964436054229736 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.2708359658718109 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.6666666865348816 jet/accuracy_1_of_6‚1 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 ;"Purity/1FRt1SRqqt1FBt/event_purity‚0.028721753507852554 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.3333333432674408 jet/accuracy_2_of_7‚0.5 7 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.9047619104385376 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.2261768877506256 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.4907407760620117 particle/accuracy_6_of_6‚1 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.4167962074279785 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 :"Purity/*FRt0SRqqt2FBt/event_purity‚0.25367599725723267 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.23851284384727478 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.1797594428062439 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.5308400392532349 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.10561291128396988 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.011136923916637897 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 particle/accuracy_1_of_4‚1 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 *"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.5 _step‚795 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9951229691505432 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 ("Purity/*FRt2SRqqt3FBt/SRqqt_purity‚1 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.3340700566768646 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.3934588134288788 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 7 Purity/3FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 :"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.48922693729400635 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 particle/accuracy_2_of_3‚1 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.10087718814611435 *"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.5 7 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.2666666805744171 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 :"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.39697855710983276 *jet/accuracy_1_of_2‚0.7366651892662048 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.4345453381538391 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 8"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.449126273393631 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9941520690917969 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.3611607253551483 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 8 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.29629629850387573 +"Purity/1FRt*SRqqt3FBt/SRqqt_purity‚0.75 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9897977113723755 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 ;"Purity/0FRt3SRqqt0FBt/event_purity‚0.026143791154026985 particle/accuracy_5_of_7‚1 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 & Purity/*FRt3SRqqt*FBt/FBt_purity‚1 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 jet/accuracy_1_of_7‚1 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.025416988879442215 :"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.42831137776374817 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.2905193269252777 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 *"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.5 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 :"Purity/*FRt*SRqqt1FBt/event_purity‚0.40240541100502014 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 jet/accuracy_5_of_5‚0 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.5714285969734192 8 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.44736841320991516 7 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.3529411852359772 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.018291493877768517 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.014624316245317459 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.4292447865009308 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 +jet/accuracy_3_of_5‚0.08961213380098343 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 & Purity/*FRt*SRqqt4FBt/FRt_purity‚0 & Purity/*FRt0SRqqt4FBt/FRt_purity‚0 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 * Purity/0FRt0SRqqt4FBt/FBt_purity‚0.875 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9896175265312195 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9826388955116272 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 jet/accuracy_5_of_7‚0 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9351851940155029 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.21281494200229645 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.6716269254684448 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 particle/accuracy_4_of_4‚1 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.3541666567325592 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 particle/accuracy_3_of_7‚1 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.4444444477558136 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.1768796443939209 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 ) Purity/3FRt1SRqqt2FBt/FRt_purity‚0.25 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.1870642751455307 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6443641185760498 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.3988667130470276 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 7 Purity/3FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.12411347776651382 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.06808905303478241 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.22727273404598236 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.3333333432674408 & Purity/0FRt3SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9267140030860901 6 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.932539701461792 7 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.2293630987405777 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.4422784745693207 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.4659722149372101 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.4577394127845764 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 *jet/accuracy_2_of_4‚0.3795573115348816 , Purity/1FRt3SRqqt1FBt/FRt_purity‚0.21875 :"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.47333332896232605 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 & Purity/0FRt2SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9930968284606934 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.4525619149208069 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9983749389648438 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.06104941666126251 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.5757575631141663 *"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.5 & Purity/1FRt*SRqqt4FBt/FRt_purity‚0 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.4836341142654419 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9942528009414673 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.6111111044883728 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 ,"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.375 ("Purity/4FRt*SRqqt0FBt/SRqqt_purity‚1 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.10393895208835602 +jet/accuracy_1_of_1‚0.44874340295791626 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.23435352742671967 :"Purity/0FRt1SRqqt2FBt/event_purity‚0.02545155957341194 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.3317834436893463 <"Purity/1FRt2SRqqt*FBt/event_purity‚0.0010582010727375746 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.26539796590805054 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.21494710445404053 +jet/accuracy_4_of_6‚0.03846153989434242 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.16631300747394562 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 @&Purity/*FRt1SRqqt3FBt/event_propPdy>Þ2ortion‚0.0006820436683483422 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9923393726348877 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9931841492652893 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.4985632300376892 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 % _timestamp‚1.7885216730399127e+09 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.4413106441497803 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 8 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.31018513441085815 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 9"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.3499999940395355 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.2698027491569519 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.12634176015853882 ) Purity/*FRt2SRqqt2FBt/FRt_purity‚0.25 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.3916613459587097 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.20000000298023224 & Purity/1FRt3SRqqt2FBt/FRt_purity‚1 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.3291351795196533 :"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.43125003576278687 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 9"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.4057988226413727 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.2848152816295624 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 7 Purity/1FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 ,jet/accuracy_3_of_3‚0.017733128741383553 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.7857142686843872 ("Purity/4FRt*SRqqt*FBt/SRqqt_purity‚1 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 particle/accuracy_1_of_1‚1 ) Purity/2FRt3SRqqt1FBt/FRt_purity‚0.25 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.21773815155029297 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.2288377285003662 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 ) Purity/4FRt0SRqqt0FBt/FRt_purity‚0.25 trainer/global_step‚9671 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.4311671555042267 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.2261768877506256 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.26425546407699585 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 particle/accuracy_4_of_6‚1 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 particle/accuracy_3_of_3‚1 particle/accuracy_2_of_4‚1 particle/accuracy_5_of_6‚1 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 +jet/accuracy_2_of_2‚0.10695594549179077 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.4374999701976776 ("Purity/0FRt2SRqqt3FBt/SRqqt_purity‚1 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 -jet/accuracy_4_of_4‚0.0011337868636474013 +particle/specificity‚0.9491193294525146 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.22478629648685455 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.9210526347160339 particle/accuracy_6_of_7‚1 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.2081618458032608 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.48656174540519714 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9969576597213745 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚1 9"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.4023568332195282 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9985979795455933 6 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.999546468257904 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.10147566348314285 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.1388888955116272 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 8 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.42716342210769653 particle/accuracy_1_of_3‚1 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9375000596046448 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 :"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.39517107605934143 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9922183752059937 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 jet/accuracy_6_of_7‚0 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 :"Purity/0FRt*SRqqt2FBt/event_purity‚0.27942755818367004 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.2411462962627411 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.34355494379997253 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.4116036593914032 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.3541666567325592 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9892512559890747 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 8"Purity/2FRt*SRqqt1FBt/event_purity‚0.027286471799016 jet/accuracy_5_of_6‚0 particle/accuracy_2_of_6‚1 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.2857142984867096 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.5690003037452698 8 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.45645979046821594 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.31041663885116577 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.3887869417667389 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 + Purity/3FRt3SRqqt0FBt/FRt_purity‚0.3125 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9896155595779419 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 *jet/accuracy_1_of_3‚0.8684682846069336 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 & Purity/*FRt3SRqqt1FBt/FBt_purity‚1 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 7 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.9166666865348816 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 ;"Purity/*FRt3SRqqt0FBt/event_purity‚0.008064515888690948 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5696281790733337 7 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5227272510528564 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.2778644859790802 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2972036302089691 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 +particle/sensitivity‚0.3591760993003845 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 *"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.5 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.3615521192550659 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.22854205965995789 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.4930046796798706 9"Purity/0FRt2SRqqt1FBt/event_purity‚0.0194968543946743 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.3143133521080017 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.0610119067132473 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.4062500298023224 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 <"Purity/1FRt2SRqqt0FBt/event_purity‚0.0012210012646391988 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.3916613459587097 7 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.1716374307870865 particle/accuracy_3_of_5‚1 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 -"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.5625 :"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.39912864565849304 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.4285714328289032 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 particle/accuracy_3_of_4‚1 7 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.2608024477958679 jet/accuracy_4_of_7‚0 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.3926376700401306 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.2801065146923065 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.1944444626569748 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8209079504013062 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 ) Purity/*FRt1SRqqt3FBt/FRt_purity‚0.75 8"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.408194363117218 :"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.40834441781044006 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.4858374297618866 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9913443922996521 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 +jet/accuracy_3_of_6‚0.11538461595773697 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 8 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.49418604373931885 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.5416666865348816 ) Purity/4FRt1SRqqt0FBt/FRt_purity‚0.25 ;"Purity/*FRt1SRqqt2FBt/event_purity‚0.013076036237180233 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5329343676567078 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.30010291934013367 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.1111111119389534 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.07822013646364212 ;"Purity/2FRt*SRqqt2FBt/event_purity‚0.023255813866853714 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 particle/accuracy_4_of_5‚1 ("Purity/2FRt*SRqqt3FBt/event_purity‚0›O±l’«(‚(:‡ ÔÙêÔîÌŸ€ gpu.0.gpu‚33.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚37.0 gpu.0.powerWatts‚114.408' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.602 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0½ “d¬(‚(:Ù ÔÙêÔ–ÊÓ€ disk.rbd2.in‚0! disk.sdb2.out‚1293.7939453125 disk.dm-0.out‚124545.5703125 disk.sdb.in‚3.1875$ memory_percent‚92.10822582244873* proc.memory.percent‚18.320035934448242 disk.rbd0.in‚0 disk.sdb1.in‚0 disk.md0.in‚ 9.26953125 disk.sda1.out‚0 disk.dm-0.in‚ 1118.4375 disk.nvme0n1.in‚ 619.63671875 network.recv‚ 3.659807e+06 disk.rbd2.out‚3.25 disk.md0p1.out‚ 1247.1875) disk./.usagePercent‚75.84081393420018 disk.sda.out‚1293.7939453125 disk.sdb1.out‚0 disk.sdb2.in‚3.1875" proc.memory.rssMB‚ 3001.5546875 network.sent‚ 2.5344582e+07$ disk./.usageGB‚2712.434860229492! disk.sda2.out‚1293.7939453125 disk.rbd0.out‚ 3.087890625 disk.md0p1.in‚ 9.26953125 disk.sda2.in‚ 6.08203125 disk.nvme1n1.in‚ 498.80078125 disk.sdb.out‚1293.7939453125) proc.memory.availableMB‚ 1292.98828125 proc.cpu.threads‚17 disk.sda.in‚ 6.08203125# disk.nvme1n1.out‚62039.24609375 disk.sda1.in‚0# disk.nvme0n1.out‚62506.32421875 cpu‚25.221522461029444 disk.md0.out‚ 1247.1875S'v§“­(‚(:ˆ ãÙêÔï²®ÿ gpu.0.gpu‚55.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚169.942' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.4855 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0—"Il®(‚(:á ãÙêÔ»’ãÿ proc.cpu.threads‚17 disk.md0.out‚ 1249.01171875 disk.sda.in‚ 6.08203125" disk.sda2.out‚1295.68701171875 disk.rbd0.in‚0$ memory_percent‚92.11142063140869" proc.memory.rssMB‚ 3001.5546875 network.sent‚ 2.5456797e+07 disk.md0.in‚ 9.26953125 disk.md0p1.out‚ 1249.01171875" disk.nvme0n1.out‚ 62545.6484375 cpu‚25.221255830716643 disk.sda2.in‚ 6.08203125 disk.sdb1.in‚0 disk.dm-0.out‚ 124690.0625! disk.sda.out‚1295.68701171875" disk.nvme1n1.out‚ 62144.4140625 disk.dm-0.in‚ 1118.4375 disk.nvme0n1.in‚ 619.63671875 disk.nvme1n1.in‚ 498.80078125) proc.memory.availableMB‚ 1292.46484375 disk.rbd2.in‚0 disk.rbd2.out‚3.25* proc.memory.percent‚18.320035934448242 disk.rbd0.out‚ 3.087890625 network.recv‚ 3.665004e+06) disk./.usagePercent‚75.84956340431066 disk.sdb1.out‚0 disk.sdb2.in‚3.1875 disk.sdb.in‚3.1875% disk./.usageGB‚2712.7477836608887" disk.sdb2.out‚1295.68701171875 disk.sda1.in‚0 disk.sda1.out‚0 disk.md0p1.in‚ 9.26953125! disk.sdb.out‚1295.68701171875[½=Í4¯(/ _wandbruntime‚11329  _runtime‚11329}vgÐÈ°( % _timestamp‚1.7885217044560416e+09 _runtime‚11329.926597467 _step‚796 & lr-AdamW/pg1‚9.819300547376132e-05 & lr-AdamW/pg2‚9.819300547376132e-05 trainer/global_step‚9699:É1Iͱ(Ç % _timestamp‚1.7885217044560416e+09 _runtime‚11329.926597467 _step‚796 & lr-AdamW/pg1‚9.819300547376132e-05 & lr-AdamW/pg2‚9.819300547376132e-05 trainer/global_step‚9699œ ¬k4²(/ _wandbruntime‚11329  _runtime‚11329À¸5(³(P R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚810482195584512 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11043001897984 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410816 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955591168 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 444356726 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 124811015 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11534402265088 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 243610278 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚810483112995328 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚894363298233344 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26088 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1212 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 226762752 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚781512598398416 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 200720360 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11043001897984 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 200720360 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚810483112995328 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚799439211689472 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚631 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚799439193686528 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1187 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚905897449795584 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 200720358 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411420 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26088 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚629 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚792514439378603 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚781512615437776 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 243636366 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177047040 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 183217418 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214665 _step‚797 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 trainer/global_step‚9699 % _timestamp‚1.7885217044571621e+09 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚810482195584512 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11043901305856 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 183216787 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 124811013 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 401960960 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚58405774 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚799439211689472 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007241 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26090 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917410816 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 243610278 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 250702848qlTžÜ´(†P QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚629 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚792514439378603 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚781512615437776 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 243636366 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177047040 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 183217418 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214665 _step‚797 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 trainer/global_step‚9699 % _timestamp‚1.7885217044571621e+09 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚810482195584512 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11043901305856 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 183216787 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 124811013 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 401960960 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚58405774 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚799439211689472 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007241 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26090 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917410816 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 243610278 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 250702848 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚810482195584512 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11043001897984 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410816 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955591168 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 444356726 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 124811015 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11534402265088 TDDeviceStatsMonitor.on_train_batch_start/a’0L0ctive_bytes.large_pool.peak‚ 12835443712 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 243610278 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚810483112995328 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚894363298233344 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26088 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1212 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 226762752 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚781512598398416 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 200720360 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11043001897984 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 200720360 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚810483112995328 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚799439211689472 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚631 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚799439193686528 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1187 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚905897449795584 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 200720358 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411420 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26088×¹@½4µ(/ _wandbruntime‚11330  _runtime‚11330!ýBp'¶(‰N _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚894452593513984 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 444375125 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚894452617454080 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11044114701312 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917424128 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11045014122496 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 183234863 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚905987936114176 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚792593528134049 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177047040 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚905988182609408 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411628 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 124823389 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11045014122496 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 222555136 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 243660983 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11044114701312 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 444401241 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚621 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚792594442585037 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007241 $ _timestamp‚1.788521705513968e+09 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955591168 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚58411474 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚799518841985024 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚623 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 243634869 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚781590578492544 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 444375125 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 200740256 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 124823391 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚799518823982080 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177047040 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 444401241 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 401960960 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 183235486 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450988 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11003847053133 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899421184 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 246495232 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚810562938683392 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 200740258 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚58412095 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11002949641505 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899421184 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11535342600192 trainer/global_step‚9699 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 _step‚798 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 200740256 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 \à'·(ŽN VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899421184 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 246495232 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚810562938683392 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 200740258 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚58412095 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11002949641505 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899421184 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11535342600192 trainer/global_step‚9699 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 _step‚798 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 200740256 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚894452593513984 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 444375125 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚894452617454080 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11044114701312 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917424128 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11045014122496 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 183234863 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚905987936114176 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚792593528134049 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177047040 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚905988182609408 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411628 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 124823389 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11045014122496 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 222555136 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 243660983 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11044114701312 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 444401241 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚621 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚792594442585037 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007241 $ _timestamp‚1.788521705513968e+09 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955591168 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚58411474 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚799518841985024 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚623 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 243634869 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚781590578492544 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 444375125 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 200740256 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 124823391 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚799518823982080 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177047040 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 444401241 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 401960960 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 183235486 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450988 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11003847053133 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0žW %4¸(/ _wandbruntime‚11330  _runtime‚11330°bB­\¹(Ö 0loss/FBt1/assignment_loss‚0.0959866107509308 0loss/SRqqt3/detection_loss‚1.485766769187259 0loss/FRt1/assignment_loss‚1.9352459535002708 2loss/SRqqt1/assignment_loss‚1.0038168730291848 2loss/SRqqt4/assignment_loss‚1.7515801255192076 1loss/SRqqt2/detection_loss‚0.8694663928218291 epoch‚8 2loss/SRqqt3/assignment_loss‚1.0550642547230902 /loss/FRt1/detection_loss‚2.4182257417830386 % _timestamp‚1.7885217055164936e+09 1loss/SRqqt4/detection_loss‚2.4690850877336095 /loss/FBt1/detection_loss‚1.6515588540779917 .loss/FRt2/detection_loss‚1.504356085932974 -loss/FRt4/detection_loss‚3.10198659658432 /loss/SRqqt1/detection_loss‚1.75695027942614 /loss/FBt4/detection_loss‚3.6972458362579346 %loss/total_loss‚37.52340911468775 trainer/global_step‚9699 0loss/FRt2/assignment_loss‚1.4295544028282166 2loss/SRqqt2/assignment_loss‚0.8096225664468849 1loss/FBt2/assignment_loss‚0.03171790419950853 .loss/FBt2/detection_loss‚0.509224063168992 _runtime‚11330.986627657 0loss/FRt3/assignment_loss‚2.0290824986166425 2loss/FBt4/assignment_loss‚0.008852737907909613 _step‚799 /loss/FRt4/assignment_loss‚2.611337251663208 1loss/FBt3/assignment_loss‚0.09009983424658685 /loss/FRt3/detection_loss‚2.4962001318732896 /loss/FBt3/detection_loss‚2.7113822624087334ØÇGÕaº(Û .loss/FRt2/detection_loss‚1.504356085932974 -loss/FRt4/detection_loss‚3.10198659658432 /loss/SRqqt1/detection_loss‚1.75695027942614 /loss/FBt4/detection_loss‚3.6972458362579346 %loss/total_loss‚37.52340911468775 trainer/global_step‚9699 0loss/FRt2/assignment_loss‚1.4295544028282166 2loss/SRqqt2/assignment_loss‚0.8096225664468849 1loss/FBt2/assignment_loss‚0.03171790419950853 .loss/FBt2/detection_loss‚0.509224063168992 _runtime‚11330.986627657 0loss/FRt3/assignment_loss‚2.0290824986166425 2loss/FBt4/assignment_loss‚0.008852737907909613 _step‚799 /loss/FRt4/assignment_loss‚2.611337251663208 1loss/FBt3/assignment_loss‚0.09009983424658685 /loss/FRt3/detection_loss‚2.4962001318732896 /loss/FBt3/detection_loss‚2.7113822624087334 0loss/FBt1/assignment_loss‚0.0959866107509308 0loss/SRqqt3/detection_loss‚1.485766769187259 0loss/FRt1/assignment_loss‚1.9352459535002708 2loss/SRqqt1/assignment_loss‚1.0038168730291848 2loss/SRqqt4/assignment_loss‚1.7515801255192076 1loss/SRqqt2/detection_loss‚0.8694663928218291 epoch‚8 2loss/SRqqt3/assignment_loss‚1.0550642547230902 /loss/FRt1/detection_loss‚2.4182257417830386 % _timestamp‚1.7885217055164936e+09 1loss/SRqqt4/detection_loss‚2.4690850877336095 /loss/FBt1/detection_loss‚1.6515588540779917Ÿœíc1d»(‚(:Ù òÙêÔ³ÓŸ disk.sdb1.in‚0 disk.sdb2.in‚3.1875 disk.md0p1.out‚ 1250.67578125 disk.nvme0n1.out‚62653.5 network.recv‚ 3.669935e+06 disk.md0.out‚ 1250.67578125" disk.sdb2.out‚1297.40673828125$ disk./.usageGB‚2713.122001647949# memory_percent‚92.1086072921753 disk.sda2.in‚ 6.08203125 disk.rbd0.out‚ 3.087890625 disk.rbd2.in‚0 disk.rbd2.out‚3.25 disk.nvme0n1.in‚ 619.63671875) proc.memory.availableMB‚ 1292.92578125! disk.nvme1n1.out‚ 62301.828125! disk.sdb.out‚1297.40673828125) disk./.usagePercent‚75.86002669585127 disk.rbd0.in‚0 disk.nvme1n1.in‚ 498.80078125 disk.sdb1.out‚0 disk.sda1.in‚0 disk.sda1.out‚0" proc.memory.rssMB‚ 3001.5546875 proc.cpu.threads‚17 network.sent‚ 2.5460872e+07" disk.sda2.out‚1297.40673828125 disk.md0.in‚ 9.26953125 disk.dm-0.in‚ 1118.4375 disk.dm-0.out‚ 124955.328125 disk.md0p1.in‚ 9.26953125 disk.sdb.in‚3.1875 disk.sda.in‚ 6.08203125! disk.sda.out‚1297.40673828125* proc.memory.percent‚18.320035934448242 cpu‚25.22095701150182ÛáõŸ“¼(‚(:ˆ òÙêÔù³Â€ gpu.0.gpu‚47.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚158.746' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚39.6865 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0çø@k‘½(‚(:† ÚêԹϳÿ gpu.0.gpu‚56.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚172.22' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.055 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0tìÆŸf¾(‚(:Û ÚêÔ• Íÿ cpu‚25.22067338992952$ disk./.usageGB‚2713.373020172119 disk.sda.out‚1298.9326171875 disk.nvme1n1.in‚ 498.80078125 network.recv‚ 3.67454e+06! disk.dm-0.out‚125183.59765625 disk.sdb.in‚3.1875 disk.sda2.in‚ 6.08203125 disk.rbd0.in‚0 disk.md0.in‚ 9.26953125 disk.dm-0.in‚ 1118.4375 disk.md0p1.in‚ 9.26953125 disk.nvme0n1.in‚ 619.63671875 disk.md0.out‚ 1252.14453125 disk.sdb2.in‚3.1875$ memory_percent‚92.11158752441406 network.sent‚ 2.5567782e+07 disk.sda.in‚ 6.08203125" disk.nvme1n1.out‚ 62422.0703125 disk.sdb1.in‚0 disk.sda1.in‚0! disk.sda2.out‚1298.9326171875 disk.sdb1.out‚0" proc.memory.rssMB‚ 3001.5546875* proc.memory.percent‚18.320035934448242 proc.cpu.threads‚17# disk.nvme0n1.out‚62761.52734375 disk.md0p1.out‚ 1252.14453125 disk.sdb.out‚1298.9326171875 disk.rbd2.out‚3.25% proc.memory.availableMB‚ 1292.4375) disk./.usagePercent‚75.86704527884646 disk.rbd0.out‚ 3.087890625 disk.rbd2.in‚0! disk.sdb2.out‚1298.9326171875 disk.sda1.out‚0åA¯ª’¿(‚(:‡ ÚêÔú¤øÿ gpu.0.gpu‚47.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚164.372' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚41.093 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0QÈÛªiÀ(‚(:Þ ÚêÔ§ÿ¾€# disk.nvme0n1.out‚62869.34765625" disk.nvme1n1.out‚ 62511.8828125 disk.md0.out‚ 1253.37890625 disk.nvme0n1.in‚ 619.63671875 proc.cpu.threads‚17) disk./.usagePercent‚75.88105343317214 disk.sda.out‚1300.232421875 disk.sda1.in‚0 disk.sda1.out‚0* proc.memory.percent‚18.320035934448242 disk.sdb.in‚3.1875 disk.rbd0.in‚0 disk.md0p1.out‚ 1253.37890625) proc.memory.availableMB‚ 1292.44921875" proc.memory.rssMB‚ 3001.5546875 disk.sda2.out‚1300.232421875 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.md0.in‚ 9.26953125 disk.sdb2.out‚1300.232421875 disk.dm-0.in‚ 1118.4375! disk.dm-0.out‚125381.23046875$ memory_percent‚92.11151599884033% disk./.usageGB‚2713.8740196228027 disk.nvme1n1.in‚ 498.80078125 disk.sdb2.in‚3.1875 disk.md0p1.in‚ 9.26953125 cpu‚25.220378839440254 disk.sdb.out‚1300.232421875 disk.sda.in‚ 6.08203125 disk.rbd2.out‚3.25 network.sent‚ 2.5576078e+07 disk.sda2.in‚ 6.08203125 network.recv‚ 3.679066e+06 disk.rbd0.out‚ 3.087890625 disk.rbd2.in‚0¡%H“Á(‚(:ˆ ŸÚêÔ‘ʽÿ gpu.0.gpu‚52.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚172.118' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.0295 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚02¨|£Â(‚(:Ù ŸÚêÔ‡½Õÿ disk.sda.in‚ 6.08203125 disk.sdb1.in‚0 disk.dm-0.in‚ 1118.4375 disk.nvme0n1.in‚ 619.63671875 disk.nvme0n1.out‚ 62869.46875" disk.sda2.out‚1302.10400390625 disk.nvme1n1.in‚ 498.80078125% disk./.usageGB‚2713.8740196228027! disk.sda.out‚1302.10400390625 disk.sda2.in‚ 6.08203125 disk.rbd0.out‚ 3.087890625# disk.nvme1n1.out‚62640.08203125 disk.Š ‹‘Ámd0p1.in‚ 9.26953125$ memory_percent‚92.11151599884033 network.recv‚ 3.68323e+06 disk.sdb2.in‚3.1875 cpu‚25.22009289893272 disk.rbd0.in‚0 disk.sda1.in‚0! disk.dm-0.out‚125509.55078125" disk.sdb2.out‚1302.10400390625) disk./.usagePercent‚75.88105343317214! disk.sdb.out‚1302.10400390625 disk.sdb1.out‚0 disk.md0.in‚ 9.26953125 disk.rbd2.out‚3.25) proc.memory.availableMB‚ 1292.44921875 disk.sdb.in‚3.1875 disk.md0.out‚ 1255.1875 disk.sda1.out‚0* proc.memory.percent‚18.320035934448242 proc.cpu.threads‚17 network.sent‚ 2.557978e+07 disk.rbd2.in‚0 disk.md0p1.out‚ 1255.1875" proc.memory.rssMB‚ 3001.5546875ïÞÕ‘4Ã(/ _wandbruntime‚11387  _runtime‚11387X –ÿÈÄ( _step‚800 & lr-AdamW/pg1‚9.428522644388083e-05 & lr-AdamW/pg2‚9.428522644388083e-05 trainer/global_step‚9749 % _timestamp‚1.7885217624219785e+09 _runtime‚11387.892443994:;=wÍÅ(Ç & lr-AdamW/pg1‚9.428522644388083e-05 & lr-AdamW/pg2‚9.428522644388083e-05 trainer/global_step‚9749 % _timestamp‚1.7885217624219785e+09 _runtime‚11387.892443994 _step‚800 ²ïòÅ4Æ(/ _wandbruntime‚11387  _runtime‚11387©$(Ç(P OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26088 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11058185048715 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚803420726467072 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 201715260 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 446558438 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚803420708464128 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 _runtime‚11387.894265932 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 244843180 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214665 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26088 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407872 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚58689867 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 _step‚801 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 244843180 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚614 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955591168 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚803420726467072 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚898829259270144 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11591939999744 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚814519364942336 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955591168 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 201715260 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11592166762496 trainer/global_step‚9749 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 125429813 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚58690479 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚898829235330048 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚785411603104816 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407872 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 125429815 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 226762752 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚796469805192891 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26090 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11098656478208 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11099555886080 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11098656478208 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 184120294 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1212 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚803420708464128 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚814519364942336 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 446584528 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1187 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚910421426032640 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11057287637295 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 184119680 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚785411620144176 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411420 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 244869268 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11099555886080 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007241 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 401960960 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177047040 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 244869268 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚612 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410816 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚910421175329792 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450780 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚796468890742111 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564›Qš (È(†P F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 446558438 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚803420708464128 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 _runtime‚11387.894265932 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 244843180 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214665 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26088 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407872 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚58689867 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 _step‚801 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 244843180 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚614 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955591168 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚803420726467072 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚898829259270144 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11591939999744 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚814519364942336 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955591168 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 201715260 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11592166762496 trainer/global_step‚9749 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 125429813 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚58690479 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚898829235330048 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚785411603104816 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407872 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 125429815 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 226762752 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚796469805192891 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26090 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11098656478208 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11099555886080 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11098656478208 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 184120294 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1212 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚803420708464128 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚814519364942336 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 446584528 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1187 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚910421426032640 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11057287637295 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 184119680 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚785411620144176 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411420 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 244869268 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11099555886080 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007241 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 401960960 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177047040 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 244869268 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚612 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410816 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚910421175329792 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450780 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚796468890742111 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26088 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11058185048715 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚803420726467072 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 201715260 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0¡ê.1å4É(/ _wandbruntime‚11388  _runtime‚11388|‘åø'Ê(ŠN PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 184138315 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 125442189 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚814601025465344 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 184137687 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚910511895811584 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 401960960 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚785489583198944 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚803500338759680 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚785489600238304 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 446629043 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚910511636733440 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚898918554550784 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 125442191 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 244893885 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11099769281536 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚796548893948545 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11058396298613 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 244867771 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899421184 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 259078144 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚628 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 446602927 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚803500338759680 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11100668702720 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11593317320704 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 446602927 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11593082182656 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007241 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177047040 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26114 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 244893885 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1187 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11059293710241 _runtime‚11388.92633772 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚814600108041216 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 446629043 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚910511636733440 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚898918554550784 RBDeviceStatsMonitor.on_train_batch_end/reseç?™Þp#rved_bytes.all.allocated‚ 14220787712 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 125442191 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 244893885 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11099769281536 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚796548893948545 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11058396298613 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 244867771 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899421184 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 259078144 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚628 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 446602927 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚803500338759680 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11100668702720 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11593317320704 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 446602927 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11593082182656 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007241 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177047040 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26114 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 244893885 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1187 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11059293710241 _runtime‚11388.92633772 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚814600108041216 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 446629043 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 125442189 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚814601025465344 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 184137687 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚910511895811584 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 401960960 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚785489583198944 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚803500338759680 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚785489600238304 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 446629043 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 PDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚818556534300160 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 246102170 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 185023657 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917410816 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 448786240 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 401960960 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚789310624850576 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11154311058432 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410816 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚800423342105619 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1212 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 202710160 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 246102170 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11650080979968 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _step‚805 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955591168 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚807402223241728 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26090 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26088 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11649856314368 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177047040 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411420 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11112734294403 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11154311058432 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚903295196366848 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450780 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 _runtime‚ 11445.868576 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚807402241244672 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚818557451710976 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚800424256556399 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 126048613 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 202710158 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 246076082 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚807402241244672 trainer/global_step‚9799 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚608 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚789310607811216 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚807402223241728 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1187 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214665 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 202710160 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚58975044 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚818557451710976 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 202710158 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007241 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚807402241244672 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚818557451710976 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚800424256556399 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 126048613 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 202710158 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 246076082 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚807402241244672 trainer/global_step‚9799 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚608 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚789310607811216 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚807402223241728 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1187 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214665 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 202710160 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚58975044 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚818557451710976 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 202710158 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007241 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚818556534300160 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 246102170 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 185023657 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917410816 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 448786240 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 401960960 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚789310624850576¥G ׈4Ý(/ _wandbruntime‚11447  _runtime‚11447*pM'Þ(‹N BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 246100673 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 448856845 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 401960960 trainer/global_step‚9799 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 448830729 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 202730058 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 448830729 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899421184 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚915035528105472 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11114740367349 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917424128 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚58981310 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26114 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11156323282944 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11155423861760 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚915035778795008 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚807481871540224 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11156323282944 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11651239267328 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450988 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚818637277399040 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚598 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 250689536 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚800503345312053 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚600 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 185041701 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0š¬Áe'ß(N TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177047040 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11651012517888 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚800502430861065 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 202730056 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚789388604944704 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 448856845 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚818638194823168 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007241 _runtime‚11447.033757763 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 126060991 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411628 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 246126787 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚789388587905344 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 % _timestamp‚1.7885218215613163e+09 _step‚806 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 246100673 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 448856845 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 401960960 trainer/global_step‚9799 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 448830729 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 202730058 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 448830729 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899421184 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚915035528105472 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11114740367349 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917424128 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚58981310 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26114 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11156323282944 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11155423861760 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚915035778795008 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚807481871540224 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11156323282944 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11651239267328 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450988 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚818637277399040 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚598 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 250689536 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚800503345312053 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚600 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 185041701 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 trainer/global_step‚9849 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚804377793469127 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚625 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955591168 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚811383756022272 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚907761181343744 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 247308984 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407872 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411420 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 _step‚809 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 203705060 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚59260313 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚793209629556976 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚793209612517616 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 126667413 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26088 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 247308984 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚822593703657984 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚919469288594944 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚907761157403648 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177047040 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚811383738019328 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11210865046528 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 246508544 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚811383756022272 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 401960960 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 % _timestamp‚1.7885218787737825e+09 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 203705060?öE (î(…P KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚59260313 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚793209629556976 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚793209612517616 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 126667413 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26088 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 247308984 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚822593703657984 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚919469288594944 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚907761157403648 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177047040 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚811383738019328 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11210865046528 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 246508544 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚811383756022272 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 401960960 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 % _timestamp‚1.7885218787737825e+09 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 203705060 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11708107251200 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11169078362931 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26088 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 _runtime‚11504.24425671 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 126667415 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚822593703657984 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 451014042 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410816 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚919469042086400 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007241 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 203705058 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚59259688 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1187 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 trainer/global_step‚9849 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚804377793469127 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚625 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955591168 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚811383756022272 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚907761181343744 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 247308984 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407872 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411420 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 _step‚809 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 203705060©\3seï(‚(:Ú —ÛêÔÈÎüþ disk.sda2.in‚ 6.08203125 proc.cpu.threads‚17 disk.sdb2.out‚ 1315.65234375 disk.md0p1.in‚ 9.26953125 disk.nvme0n1.in‚ 619.63671875 disk.sdb.in‚3.1875# disk.nvme1n1.out‚64223.27734375 disk.md0.out‚ 1268.25390625) disk./.usagePercent‚75.87402823722394 disk.nvme1n1.in‚ 498.80078125 disk.sda1.out‚0 disk.sda.in‚ 6.08203125 disk.sda2.out‚ 1315.65234375 network.recv‚ 3.720408e+06) proc.memory.availableMB‚ 1291.82421875" proc.memory.rssMB‚ 3001.5546875 cpu‚25.218044000232616 disk.dm-0.in‚ 1118.4375 disk.md0p1.out‚ 1268.25390625# disk.nvme0n1.out‚63111.62109375$ memory_percent‚92.11533069610596 disk.dm-0.out‚127334.8984375 disk.rbd0.in‚0 disk.rbd0.out‚ 3.087890625* proc.memory.percent‚18.320035934448242 network.sent‚ 2.5830611e+07 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.md0.in‚ 9.26953125 disk.rbd2.in‚0 disk.rbd2.out‚3.25 disk.sda1.in‚0 disk.sdb.out‚ 1315.65234375% disk./.usageGB‚2713.6227645874023 disk.sdb2.in‚3.1875 disk.sda.out‚ 1315.65234375}×7”ð(‚(:‰ —ÛêÔûó¥€ gpu.0.gpu‚38.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚171.777' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.94425 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÿÒr4ñ(/ _wandbruntime‚11505  _runtime‚11505ãó#ò(ˆN REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 451084647 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11169289612829 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 trainer/global_step‚9849 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 247333575 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11709045977600 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 451058531 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 203724958 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11709270629888 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚822675364180992 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 224652288 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11170187024457 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 247359689 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 401960960 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955591168 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚811463368314880 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚619 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚919559771194368 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 203724956 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177047040 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚59265262 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411628 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11211977863168 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 203724956 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 248592384 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚822675364180992 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917424128 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899421184 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 $ _timestamp‚1.788521879759142e+09 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚811463386317824 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚811463368314880 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚907850500564480 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26114 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899421184 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 247333575 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 _step‚810 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 451084647 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 185945051 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007241 _runtime‚ 11505.2300022ô{Ò'ó(N TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177047040 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚907850476624384 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 451084647 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 185945051 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007241 _runtime‚ 11505.2300022 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 451084647 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11169289612829 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 trainer/global_step‚9849 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 247333575 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11709045977600 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 451058531 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 203724958 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11709270629888 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚822675364180992 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 224652288 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11170187024457 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 247359689 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 401960960 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955591168 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚811463368314880 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚619 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚919559771194368 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 203724956 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177047040 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚59265262 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411628 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11211977863168 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 203724956 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 248592384 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚822675364180992 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917424128 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899421184 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 $ _timestamp‚1.788521879759142e+09 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚811463386317824 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚811463368314880 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚907850500564480 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26114 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899421184 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 247333575 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 _step‚810 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚826630873015808 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11224525020039 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚11562.695669051 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 204699958 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 401960960 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚808332244832635 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚797108634263376 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚826631790426624 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11266519626752 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 204699958 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚59546082 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 186831700 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚808333159283415 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 trainer/global_step‚9899 % _timestamp‚1.7885219372251961e+09 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚815365270799872 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚826630873015808 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 218374144 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411420 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410816 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 204699960 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚826631790426624 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚597 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚923993349749248 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 453241844 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11266519626752 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 204699960 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11765988994560 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 248541886 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 248567974 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚595 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214665 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 248567974 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407872 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955591168 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11766207368704 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177047040 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955591168 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917410816 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11265620218880 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚815365252796928 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚815365270799872 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26088 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1187 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚923993107435008 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚912227118440448 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 453267934 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 127286213 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚912227142380544 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007241 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26090 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _step‚813 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚815365252796928 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11265620218880 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 242314240 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚797108617224016 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11223627608619 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26088 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 248541886µ¤s” ‚)†P R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚826630873015808 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11224525020039 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚11562.695669051 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 204699958 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 401960960 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚808332244832635 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚797108634263376 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚826631790426624 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11266519626752 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 204699958 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚59546082 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 186831700 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚808333159283415 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 trainer/global_step‚9899 % _timestamp‚1.7885219372251961e+09 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚815365270799872 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚826630873015808 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 218374144 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411420 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410816 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ?9DeviceStatsˆ”ÔWxMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 204699960 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚826631790426624 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚597 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚923993349749248 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 453241844 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11266519626752 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 204699960 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11765988994560 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 248541886 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 248567974 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚595 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214665 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 248567974 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407872 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955591168 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11766207368704 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177047040 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955591168 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917410816 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11265620218880 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚815365252796928 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚815365270799872 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26088 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1187 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚923993107435008 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚912227118440448 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 453267934 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 127286213 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚912227142380544 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007241 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26090 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _step‚813 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚815365252796928 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11265620218880 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 242314240 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚797108617224016 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11223627608619 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26088 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 248541886 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407872 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚59545487 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0­–_Öë4ƒ)/ _wandbruntime‚11563  _runtime‚11563R -n'„)ŠN ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 127298589 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚592 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚924083574745088 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 230943744 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899421184 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007241 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177047040 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899421184 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 204719858 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚912316437661184 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11767368027648 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 _step‚814 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚815444883092480 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 453312449 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 186849677 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917424128 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 401960960 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955591168 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚826712533538816 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 186850271 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚59551088 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1187 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚815444883092480 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11767137083904 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 204719856 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 254883840 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 248592591 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚815444901095424 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11266733022208 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚59551680 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11266733022208 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _runtime‚11563.762116638 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 $ _timestamp‚1.788521938289931e+09 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177047040 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚924083829628928 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607|“i¾'…)N H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 453312449 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 204719856 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31686 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚797186614357504 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 248566477 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚59551088 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1187 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚815444883092480 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11767137083904 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 204719856 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 254883840 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 248592591 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚815444901095424 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11266733022208 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚59551680 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11266733022208 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _runtime‚11563.762116638 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 $ _timestamp‚1.788521938289931e+09 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177047040 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚924083829628928 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 127298589 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚592 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚924083574745088 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 230943744 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899421184 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007241 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177047040 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899421184 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 204719858 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚912316437661184 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11767368027648 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 _step‚814 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚815444883092480 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 453312449 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 186849677 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917424128 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 401960960 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955591168 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚826712533538816 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 186850271 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚830668042373632 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177047040 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1212 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11279971677147 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11279074265727 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚819346767574528 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚801007621930416 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 455469646 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚830668959784448 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 249800876 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 127905013 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 205694860 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 205694858 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917410816 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955591168 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11321274799104 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007241 trainer/global_step‚9949 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚812286696196143 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26088 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚625 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1187 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚801007638969776 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 249800876 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚59832099 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚916693079477248 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _runtime‚11617.23255083 à (–)…P VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 252800000 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚830668042373632 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 249774788 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407872 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 % _timestamp‚1.7885219917618816e+09 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚830668042373632 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177047040 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1212 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11279971677147 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11279074265727 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚819346767574528 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚801007621930416 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 455469646 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚830668959784448 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 249800876 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 127905013 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 205694860 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 205694858 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917410816 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955591168 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11321274799104 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007241 trainer/global_step‚9949 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚812286696196143 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26088 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚625 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1187 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚801007638969776 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 249800876 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚59832099 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚916693079477248 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _runtime‚11617.23255083 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 205694858 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚819346785577472 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955591168 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚623 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚916693103417344 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11824175652864 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚830668959784448 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26090 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 455495736 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚928517279070208 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214665 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚819346767574528 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407872 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚59831476 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 J÷4—)/ _wandbruntime‚11618  _runtime‚11618]úèa!˜)‹N MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1187 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚819426415873024 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚928607747970560 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 205714758 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚604 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26114 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚812365784951589 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚830749702896640 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚819426415873024 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 127917391 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450988 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411628 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 % _timestamp‚1.7885219927422097e+09 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 249825493 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 455540251 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 205714756 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚59837131 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚916782422638080 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 187754520 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26114 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955591168 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚916782398697984 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 205714756 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚830749702896640 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11825096485888 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚9949 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 249799379 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007241 _step‚818 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 249799379 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 187755124 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917424128 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚830748785472512 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177047040 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177047040 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚602 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚812366699402577 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 Å—'™)N _runtime‚11618.212978438 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 401960960 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 228846592 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11323287023616 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11280182927045 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177047040 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177047040 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚602 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚812366699402577 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1187 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚819426415873024 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚928607747970560 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 205714758 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚604 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26114 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚812365784951589 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚830749702896640 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚819426415873024 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 127917391 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450988 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411628 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 % _timestamp‚1.7885219927422097e+09 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 249825493 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 455540251 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 205714756 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚59837131 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚916782422638080 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 187754520 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26114 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955591168 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚916782398697984 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 205714756 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚830749702896640 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11825096485888 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚9949 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 249799379 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007241 _step‚818 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 249799379 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 187755124 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917424128 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚830748785472512 FDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚834705211731456 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚921159040514048 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11377828787200 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 206689758 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚933041289293312 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11882224839168 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 207888384 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚804906643676176 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26088 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 188640905 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11882016950784 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26090 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚60117680 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚834706129142272 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 251007690 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚823328300355072 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407872 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 401960960 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11376929379328 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1187 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚816241147559651 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11334520922835 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚823328282352128 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 251033778 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007241 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 457697448 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11376929379328 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450780 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917410816 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚823328282352128 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11335418334255 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 206689758 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚60117092 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚804906626636816 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 % _timestamp‚1.7885220458994923e+09 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 128523813 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11377828787200 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407872 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1212 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0Žo¯ŒÊª)†P LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407872 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1212 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚834705211731456 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚816242062010431 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214665 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 251033778 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26088 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚834706129142272 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚834705211731456 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚921159040514048 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11377828787200 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 206'yOB"689758 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚933041289293312 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11882224839168 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 207888384 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚804906643676176 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26088 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 188640905 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11882016950784 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26090 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚60117680 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚834706129142272 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 251007690 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚823328300355072 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407872 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 401960960 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11376929379328 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1187 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚816241147559651 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11334520922835 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚823328282352128 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 251033778 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007241 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 457697448 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11376929379328 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450780 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917410816 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚823328282352128 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11335418334255 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 206689758 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚60117092 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚804906626636816 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 % _timestamp‚1.7885220458994923e+09 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 128523813 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11377828787200µP¾1µ4«)/ _wandbruntime‚11672  _runtime‚11672õèšB'¬)‹N D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 251058395 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 128536191 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 226749440 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917424128 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11883171860992 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚823407930650624 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚823407930650624 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚933131531595776 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450988 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31686 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 457768053 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 206709656 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 206709656 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚834786872254464 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955591168 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11378941603840 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚60122734 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1187 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177047040 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11378941603840 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚598 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚834786872254464 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚823407912647680 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11336526995781 trainer/global_step‚9999 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 457768053 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚816320236315097 % _timestamp‚1.7885220468747625e+09 _runtime‚11672.345887333 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 457741937 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚921248383674880 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 457741937 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26114 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚804984606730944 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚834785954830336 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 250689536 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899421184 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 206709658 _step‚822 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚921248359734784 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 251032281 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 188658923 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411628 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899421184 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚804984623770304 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11378042182656 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450988 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31686 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 457768053 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 206709656 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 206709656 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚834786872254464 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955591168 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11378941603840 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚60122734 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1187 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177047040 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11378941603840 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚598 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚834786872254464 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚823407912647680 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11336526995781 trainer/global_step‚9999 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 457768053 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚816320236315097 % _timestamp‚1.7885220468747625e+09 _runtime‚11672.345887333 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 457741937 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚921248383674880 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 457741937 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26114 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚804984606730944 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚834785954830336 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 250689536 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899421184 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 206709658 _step‚822 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚921248359734784 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 251032281 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 188658923 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411628 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899421184 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚804984623770304 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11378042182656 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 251058395 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 128536191 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 226749440 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917424128 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11883171860992 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚823407930650624 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚823407930650624 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116¶ á4®)/ _wandbruntime‚11672  _runtime‚11672S/ Ob¯)Ü 1loss/FBt2/assignment_loss‚0.14156107825528583 1loss/FBt3/assignment_loss‚0.21027659201465873 0loss/FBt4/assignment_loss‚0.3089321130385656 1loss/SRqqt3/detection_loss‚1.4905579776010092 /loss/FBt4/detection_loss‚2.1885195784270763 .loss/FRt3/detection_loss‚2.393545414942006 _step‚823 0loss/FRt3/assignment_loss‚1.8044660190741222 0loss/FRt4/assignment_loss‚2.8623299815438012 2loss/SRqqt2/assignment_loss‚0.9539507725022056 .loss/FRt1/detection_loss‚1.917563215373084 /loss/FRt2/detection_loss‚1.5246589632769083 /loss/FBt1/detection_loss‚3.8494770899415016 2loss/SRqqt3/assignment_loss‚1.0753545858988576 /loss/FRt4/detection_loss‚3.2789150042967363 1loss/SRqqt2/detection_loss‚1.0332876969128848 /loss/FBt2/detection_loss‚0.5814171176199324 trainer/global_step‚9999 0loss/FRt2/assignment_loss‚1.1796356007970612 2loss/SRqqt4/assignment_loss‚1.6842102825641632 &loss/total_loss‚39.397394195240416 epoch‚8 2loss/SRqqt1/assignment_loss‚1.1992172572161939 1loss/SRqqt1/detection_loss‚1.6236099699291133 1loss/SRqqt4/detection_loss‚2.9256152123212815 /loss/FBt3/detection_loss‚2.1812093528834255 0loss/FRt1/assignment_loss‚2.7923895621299746 % _timestamp‚1.7885220468777022e+09 _runtime‚11672.350801074 1loss/FBt1/assignment_loss‚0.19669375668056924Ùæ*g°)á 0loss/FRt3/assignment_loss‚1.8044660190741222 0loss/FRt4/assignment_loss‚2.8623299815438012 2loss/SRqqt2/assignment_loss‚0.9539507725022056 .loss/FRt1/detection_loss‚1.917563215373084 /loss/FRt2/detection_loss‚1.5246589632769083 /loss/FBt1/detection_loss‚3.8494770899415016 2loss/SRqqt3/assignment_loss‚1.0753545858988576 /loss/FRt4/detection_loss‚3.2789150042967363 1loss/SRqqt2/detection_loss‚1.0332876969128848 /loss/FBt2/detection_loss‚0.5814171176199324 trainer/global_step‚9999 0loss/FRt2/assignment_loss‚1.1796356007970612 2loss/SRqqt4/assignment_loss‚1.6842102825641632 &loss/total_loss‚39.397394195240416 epoch‚8 2loss/SRqqt1/assignment_loss‚1.1992172572161939 1loss/SRqqt1/detection_loss‚1.6236099699291133 1loss/SRqqt4/detection_loss‚2.9256152123212815 /loss/FBt3/detection_loss‚2.1812093528834255 0loss/FRt1/assignment_loss‚2.7923895621299746 % _timestamp‚1.7885220468777022e+09 _runtime‚11672.350801074 1loss/FBt1/assignment_loss‚0.19669375668056924 1loss/FBt2/assignment_loss‚0.14156107825528583 1loss/FBt3/assignment_loss‚0.21027659201465873 0loss/FBt4/assignment_loss‚0.3089321130385656 1loss/SRqqt3/detection_loss‚1.4905579776010092 /loss/FBt4/detection_loss‚2.1885195784270763 .loss/FRt3/detection_loss‚2.393545414942006 _step‚823· )ð’±)‚(:‡ ËÜêÔäáÓÿ gpu.0.gpu‚26.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚169.77' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.4425 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0½¹¸•²)‚(:Í ËÜêÔÔ™¸‚) proc.memory.availableMB‚ 1292.17578125 disk.rbd2.out‚3.25 disk.sda1.in‚0 disk.sdb.in‚3.1875 proc.cpu.threads‚17 disk.sda2.in‚ 6.08203125 disk.rbd0.out‚ 3.087890625) disk./.usagePercent‚75.85305824330617 disk.nvme1n1.in‚ 498.80078125 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.rbd2.in‚0 disk.dm-0.in‚ 1118.4375! disk.dm-0.out‚129235.16796875 disk.md0p1.in‚ 9.26953125 network.recv‚ 3.774524e+06 disk.md0.in‚ 9.26953125 disk.nvme0n1.in‚ 619.63671875 cpu‚25.21474085390849$ disk./.usageGB‚2712.872776031494 disk.sda.in‚ 6.08203125 disk.sdb2.in‚3.18Ö=·Ã75! disk.sdb2.out‚1336.0986328125$ memory_percent‚92.11318492889404* proc.memory.percent‚18.320035934448242 disk.sda.out‚1336.0986328125! disk.sda2.out‚1336.0986328125# disk.nvme1n1.out‚65518.52734375 disk.sda1.out‚0 disk.sdb.out‚1336.0986328125 disk.rbd0.in‚0" proc.memory.rssMB‚ 3001.5546875 network.sent‚ 2.6298466e+07 disk.md0.out‚1288 disk.md0p1.out‚1288! disk.nvme0n1.out‚ 63716.640625ºK/”³)‚(:‰ ÚÜêÔ«àÐÿ gpu.0.gpu‚24.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚178.629' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚44.65725 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0-7©[´)‚(:Ð ÚÜêÔ–”Š€ disk.md0.in‚ 9.26953125 proc.cpu.threads‚17 disk.sda2.out‚1337.595703125 disk.rbd0.out‚ 3.087890625 network.recv‚ 3.78066e+06% disk./.usageGB‚2712.9352531433105 disk.sda.in‚ 6.08203125 disk.dm-0.in‚ 1118.4375* proc.memory.percent‚18.320035934448242 network.sent‚ 2.6393573e+07 disk.nvme0n1.in‚ 619.63671875 disk.sda.out‚1337.595703125 disk.sda2.in‚ 6.08203125 disk.dm-0.out‚ 129266.4375 disk.md0p1.in‚ 9.26953125" proc.memory.rssMB‚ 3001.5546875 cpu‚25.214412827050882 disk.rbd0.in‚0 disk.nvme1n1.in‚ 498.80078125 disk.nvme1n1.out‚ 65518.6875 disk.sdb1.in‚0) disk./.usagePercent‚75.85480512950124 disk.sdb1.out‚0 disk.rbd2.out‚3.25 disk.sda1.in‚0$ memory_percent‚92.11227893829346 disk.sdb2.in‚3.1875 disk.sdb2.out‚1337.595703125 disk.sda1.out‚0 disk.md0p1.out‚ 1289.44140625) proc.memory.availableMB‚ 1292.32421875 disk.sdb.out‚1337.595703125 disk.md0.out‚ 1289.44140625 disk.rbd2.in‚0 disk.nvme0n1.out‚63754.25 disk.sdb.in‚3.1875ÌPA_‘µ)‚(:† éÜêÔ™ÓÚÿ gpu.0.gpu‚7.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚176.892' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚44.223 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 !±e¶)‚(:Ú éÜêÔ¶Ûô! disk.sdb2.out‚1339.0126953125$ memory_percent‚92.11235046386719 network.sent‚ 2.6401005e+07" proc.memory.rssMB‚ 3001.5546875 disk.md0p1.in‚ 9.26953125 disk.nvme0n1.in‚ 619.63671875 disk.sdb.out‚1339.0126953125 disk.sda.in‚ 6.08203125 disk.rbd0.out‚ 3.087890625 disk.sdb1.in‚0% proc.memory.availableMB‚ 1292.3125 proc.cpu.threads‚17 disk.md0.out‚ 1290.8046875 disk.sda1.in‚0 disk.sdb1.out‚0% disk./.usageGB‚2713.1234817504883 disk.sdb2.in‚3.1875 disk.md0p1.out‚ 1290.8046875 disk.sda.out‚1339.0126953125# disk.nvme1n1.out‚65518.97265625* proc.memory.percent‚18.320035934448242) disk./.usagePercent‚75.86006808013772 disk.sda1.out‚0! disk.sda2.out‚1339.0126953125 disk.md0.in‚ 9.26953125 cpu‚25.214109403020004 network.recv‚ 3.78399e+06 disk.dm-0.in‚ 1118.4375! disk.dm-0.out‚129503.83203125! disk.nvme0n1.out‚ 63984.859375 disk.sdb.in‚3.1875 disk.rbd0.in‚0 disk.nvme1n1.in‚ 498.80078125 disk.sda2.in‚ 6.08203125 disk.rbd2.in‚0 disk.rbd2.out‚3.25-Ýj4·)/ _wandbruntime‚11724  _runtime‚11724Üÿ.âɸ)à _step‚824 & lr-AdamW/pg1‚7.233020909984054e-05 & lr-AdamW/pg2‚7.233020909984054e-05 trainer/global_step‚10049 % _timestamp‚1.7885220991056669e+09 _runtime‚11724.576005241aÃι)È & lr-AdamW/pg2‚7.233020909984054e-05 trainer/global_step‚10049 % _timestamp‚1.7885220991056669e+09 _runtime‚11724.576005241 _step‚824 & lr-AdamW/pg1‚7.233020909984054e-05¸­oêÙ4º)/ _wandbruntime‚11724  _runtime‚11724÷ ±5(»)P N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 189546703 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955591168 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26088 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚838743298500096 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1187 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11390864991363 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407872 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177047040 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚838742381089280 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚827309797129728 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11432583959552 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚60404090 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11940060029440 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11433483367424 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11433483367424 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407872 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 189547330 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 129142615 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚827309815132672 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917410816 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚937565061580288 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚10049 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚827309797129728 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 _runtime‚11724.577165594 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 129142613 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚838742381089280 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚820195598923159 _step‚825 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955591168 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚60404715 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 207684658 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 252240592 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚925625025490944 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚808805648382576 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 207684660 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚627 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26090 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 224665600 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 207684660 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 207684658 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚925625001550848 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410816 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚827309815132672 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11940284695040 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 $ _timestamp‚1.788522099106702e+09 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0ôÅV (¼)†P HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11433483367424 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11433483367424 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407872 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 189547330 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 129142615 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚827309815132672 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917410816 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚937565061580288 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚10049 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚827309797129728 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 _runtime‚11724.577165594 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 129142613 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚838742381089280 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚820195598923159 _step‚825 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955591168 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚60404715 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 207684658 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 252240592 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚925625025490944 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚808805648382576 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 207684660 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚627 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26090 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 224665600 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 207684660 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 207684658 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚925625001550848 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410816 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚827309815132672 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11940284695040 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 $ _timestamp‚1.788522099106702e+09 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 189546703 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955591168 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26088 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚838743298500096 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1187 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11390864991363 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407872 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177047040 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚838742381089280 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚827309797129728 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11432583959552 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚60404090 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11940060029440 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0¹Ÿ *4½)/ _wandbruntime‚11725  _runtime‚11725Ü\·¾)ŒN TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 263272448 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007241 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 252291297 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 252265183 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899421184 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450988 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚625 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 401960960 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚827389445428224 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚937655516815360 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11941435376128 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899421184 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917424128 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177047040 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 207704558 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1187 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚838823124188160 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 trainer/global_step‚10049 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 459995855 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 189565406 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚827389427425280 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31686 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚808883611437344 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 252265183 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 129154991 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 207704556 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚838824041612288 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 189564781 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚838824041612288 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚827389427425280 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26114 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 189564781 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚838824041612288 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚827389427425280 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26114 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917424128 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177047040 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 207704558 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1187 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚838823124188160 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 trainer/global_step‚10049 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 459995855 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 189565406 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚827389427425280 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31686 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚808883611437344 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 252265183 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 129154991 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 207704556 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚838824041612288ºþ»µð4À)/ _wandbruntime‚11725  _runtime‚11725ƒt#ŠeÁ)ß 2loss/SRqqt2/assignment_loss‚0.8161293631031489 1loss/FBt2/assignment_loss‚0.08225807373549832 /loss/FRt3/detection_loss‚2.2301268711453304 .loss/FRt4/detection_loss‚3.364023658362302 /loss/FRt1/assignment_loss‚2.162705493600745 /loss/FRt2/detection_loss‚1.5802056503062154 1loss/SRqqt3/detection_loss‚1.8795701883394609 % _timestamp‚1.7885221001613195e+09 _step‚827 0loss/FRt2/assignment_loss‚1.3229357752145505 0loss/FRt4/assignment_loss‚3.7728253494609487 1loss/SRqqt1/assignment_loss‚1.027984744728063 2loss/SRqqt3/assignment_loss‚1.5752619315065781 2loss/FBt4/assignment_loss‚0.006617621514487837 /loss/FBt1/detection_loss‚2.1813157374660173 0loss/FBt2/detection_loss‚0.36790858340021726 0loss/FRt3/assignment_loss‚3.1347264175613723 1loss/FBt1/assignment_loss‚0.06131929353068699 epoch‚8 _runtime‚11725.631475682 /loss/FRt1/detection_loss‚1.8245054866568018 1loss/SRqqt1/detection_loss‚1.2925263777600997 /loss/FBt3/detection_loss‚2.1550774693489076 2loss/SRqqt4/assignment_loss‚1.4464482538402081 1loss/SRqqt2/detection_loss‚0.8198264719297489 1loss/SRqqt4/detection_loss‚2.5223180770874025 /loss/FBt4/detection_loss‚3.7984872102737426 &loss/total_loss‚39.555350723127546 1loss/FBt3/assignment_loss‚0.13024662325501596 trainer/global_step‚10049/9…jÂ)ä 0loss/FRt4/assignment_loss‚3.7728253494609487 1loss/SRqqt1/assignment_loss‚1.027984744728063 2loss/SRqqt3/assignment_loss‚1.5752619315065781 2loss/FBt4/assignment_loss‚0.006617621514487837 /loss/FBt1/detection_loss‚2.1813157374660173 0loss/FBt2/detection_loss‚0.36790858340021726 0loss/FRt3/assignment_loss‚3.1347264175613723 1loss/FBt1/assignment_loss‚0.06131929353068699 epoch‚8 _runtime‚11725.631475682 /loss/FRt1/detection_loss‚1.8245054866568018 1loss/SRqqt1/detection_loss‚1.2925263777600997 /loss/FBt3/detection_loss‚2.1550774693489076 2loss/SRqqt4/assignment_loss‚1.4464482538402081 1loss/SRqqt2/detection_loss‚0.8198264719297489 1loss/SRqqt4/detection_loss‚2.5223180770874025 /loss/FBt4/detection_loss‚3.7984872102737426 &loss/total_loss‚39.555350723127546 1loss/FBt3/assignment_loss‚0.13024662325501596 trainer/global_step‚10049 2loss/SRqqt2/assignment_loss‚0.8161293631031489 1loss/FBt2/assignment_loss‚0.08225807373549832 /loss/FRt3/detection_loss‚2.2301268711453304 .loss/FRt4/detection_loss‚3.364023658362302 /loss/FRt1/assignment_loss‚2.162705493600745 /loss/FRt2/detection_loss‚1.5802056503062154 1loss/SRqqt3/detection_loss‚1.8795701883394609 % _timestamp‚1.7885221001613195e+09 _step‚827 0loss/FRt2/assignment_loss‚1.3229357752145505»ç”Ã)‚(:‰ øÜêÔ‡í€ gpu.0.gpu‚55.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚37.0 gpu.0.powerWatts‚122.153' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚30.53825 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Åo:._Ä)‚(:Ô øÜêÔIJˆƒ% disk./.usageGB‚2712.8436393737793 disk.dm-0.in‚ 1118.4375 disk.sda2.out‚1341.033203125 disk.sdb2.in‚3.1875& proc.memory.availableMB‚ 1292.28125 disk.sda1.in‚0 disk.rbd0.in‚0 disk.rbd2.in‚0 disk.sdb.in‚3.1875 disk.rbd0.out‚ 3.087890625# disk.nvme1n1.out‚65519.14453125 disk.rbd2.out‚3.25) disk./.usagePercent‚75.85224357016182 disk.sdb.out‚1341.033203125 disk.md0.in‚ 9.26953125$ memory_percent‚92.11254119873047* proc.memory.percent‚18.320035934448242 disk.sda1.out‚0 disk.sda.in‚ 6.08203125 disk.sdb1.in‚0 disk.md0p1.in‚ 9.26953125 disk.md0p1.out‚ 1292.765625 disk.nvme0n1.in‚ 619.63671875 disk.sda2.in‚ 6.08203125 disk.sdb1.out‚0 network.sent‚ 2.6404846e+07# disk.nvme0n1.out‚64202.66796875 disk.sda.out‚1341.033203125 disk.nvme1n1.in‚ 498.80078125 disk.md0.out‚ 1292.765625 disk.sdb2.out‚1341.033203125" proc.memory.rssMB‚ 3001.5546875 cpu‚25.213788424624717 network.recv‚ 3.788281e+06 disk.dm-0.out‚ 129721.8125 proc.cpu.threads‚177V¸x’Å)‚(:‡ ‡ÝêÔºŽ¼ÿ gpu.0.gpu‚26.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚163.09' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚40.7725 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0õ¢ÇaÆ)‚(:Ö ‡ÝêÔ’ˆ€ disk.md0.out‚ 1294.7421875 disk.sdb2.in‚3.1875 disk.sda1.in‚0 disk.sdb.in‚3.1875" proc.memory.rssMB‚ 3001.5546875 disk.rbd2.out‚3.25! disk.dm-0.out‚129723.16015625 disk.sda2.in‚ 6.08203125* proc.memory.percent‚18.320035934448242 proc.cpu.threads‚17 disk.rbd2.in‚0 disk.sda.in‚ 6.08203125) proc.memory.availableMB‚ 1292.05078125 network.sent‚ 2.6512532e+07% disk./.usageGB‚2712.8448181152344 disk.dm-0.in‚ 1118.4375# disk.nvme0n1.out‚64203.26953125 disk.rbd0.in‚0 disk.nvme1n1.in‚ 498.80078125$ memory_percent‚92.11394786834717 disk.md0p1.in‚ 9.26953125 disk.sdb.out‚ 1343.07421875 disk.rbd0.out‚ 3.087890625! disk.nvme1n1.out‚ 65519.890625 disk.sdb1.in‚0 disk.sdb1.out‚0 network.recv‚ 3.793878e+06 disk.sda1.out‚0 disk.md0p1.out‚ 1294.7421875 disk.nvme0n1.in‚ 619.63671875 cpu‚25.21344430788326) disk./.usagePercent‚75.85227652826624 disk.md0.in‚ 9.26953125 disk.sdb2.out‚ 1343.07421875 disk.sda.out‚ 1343.07421875 disk.sda2.out‚ 1343.07421875+?TÖ”Ç)‚(:‰ –ÝêÔ¼Ýëý gpu.0.gpu‚48.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚157.977' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚39.49425 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Nȹ jÈ)‚(:ß –ÝêÔ‡«›€ network.recv‚ 3.798614e+06 disk.sdb.in‚3.1875! disk.sdb.out‚1344.49267578125 disk.rbd2.in‚0 network.sent‚ 2.6521032e+07) disk./.usagePercent‚75.86448073301213 disk.nvme1n1.in‚ 498.80078125 disk.rbd2.out‚3.25 disk.sda1.in‚0 disk.md0p1.in‚ 9.26953125 disk.rbd0.in‚0" proc.memory.rssMB‚ 3001.5546875 disk.sdb1.out‚0 disk.md0.in‚ 9.26953125# disk.nvme0n1.out‚64570.91015625 disk.sda.in‚ 6.08203125! disk.sda.out‚1344.49267578125" disk.sda2.out‚1344.49267578125 disk.rbd0.out‚ 3.087890625# disk.nvme1n1.out‚65520.12890625 disk.sdb1.in‚0" disk.sdb2.out‚1344.49267578125 disk.dm-0.in‚ 1118.4375 disk.md0p1.out‚ 1296.09375$ memory_percent‚92.11583137512207( proc.memory.availableMB‚ 1291.7421875 disk.sda1.out‚0 disk.nvme0n1.in‚ 619.63671875 disk.sda2.in‚ 6.08203125* proc.memory.percent‚18.320035934448242% disk./.usageGB‚2713.2812995910645 disk.md0.out‚ 1296.09375 disk.sdb2.in‚3.1875 disk.dm-0.out‚130091.0390625 cpu‚25.213150975108395 proc.cpu.threads‚17,ç#”É)‚(:‰ ¥ÝêÔ‚ò›ÿ gpu.0.gpu‚47.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚37.0 gpu.0.powerWatts‚149.165' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚37.29125 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ú£˜ÖpÊ)‚(:å ¥ÝêÔ¼ž°€ disk.sda2.in‚ 10.7578125 disk.dm-0.out‚ 130183.234375# disk.nvme0n1.out‚64662.83984375% disk./.usageGB‚2713.3709716796875 disk.rbd2.out‚3.25 disk.nvme0n1.in‚ 619.63671875 proc.cpu.threads‚17 network.sent‚ 2.6526432e+07) disk./.usagePercent‚75.86698800214073" disk.sda2.out‚1346.97216796875 disk.rbd2.in‚0! disk.sdb.out‚1346.97216796875) proc.memory.availableMB‚ 1291.98828125 cpu‚25.212921659773304 disk.sda1.in‚0 disk.md0p1.out‚ 1298.51171875 disk.sdb1.out‚0 network.recv‚ 3.802397e+06 disk.md0.out‚ 1298.51171875" disk.sdb2.out‚1346.97216796875 disk.rbd0.out‚ 3.087890625 disk.nvme1n1.in‚ 498.80078125 disk.md0.in‚17.8125 disk.sdb.in‚ 7.0546875 disk.md0p1.in‚17.8125! disk.sda.out‚1346.97216796875# disk.nvme1n1.out‚65520.39453125 disk.sdb1.in‚0 disk.sdb2.in‚ 7.0546875$ memory_percent‚92.11432933807373" proc.memory.rssMB‚ 3001.5546875 disk.sda.in‚ 10.7578125 disk.rbd0.in‚0 disk.sda1.out‚0 disk.dm-0.in‚ 1118.4375* proc.memory.percent‚18.320035934448242í­ê4Ë)/ _wandbruntime‚11781  _runtime‚11781üBí}ÉÌ)à & lr-AdamW/pg1‚6.892676678709668e-05 & lr-AdamW/pg2‚6.892676678709668e-05 trainer/global_step‚10099 % _timestamp‚1.7885221555796356e+09 _runtime‚11781.050156887 _step‚828u@í8ÎÍ)È & lr-AdamW/pg2‚6.892676678709668e-05 trainer/global_step‚10099 % _timestamp‚1.7885221555796356e+09 _runtime‚11781.050156887 _step‚828 & lr-AdamW/pg1‚6.892676678709668e-05¼"ÜŠ¾4Î)/ _wandbruntime‚11781  _runtime‚11781Î¥ÜE(Ï)‚P E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚625 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚623 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 401960960 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚824150964737447 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26088 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚831291329910272 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007241 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11488238539776 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1212 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 190452617 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚942089179929600 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 208679560 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 129761413 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚60691202 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11445414237051 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _runtime‚11781.051662404 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚831291311907328 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚812704653088976 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407872 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚842780467857920 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 253499582 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 226762752 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11998217341952 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 462153052 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11488238539776 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚824150050286667 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚812704636049616 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11446311648471 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411420 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _step‚829 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚842779550447104 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450780 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410816 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚831291311907328 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 129761415 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955591168 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚942089430632448 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214665 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 208679558 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1187 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 253473494 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11489137947648 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚930090962587648 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚842779550447104 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407872 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 208679558 trainer/global_step‚10099 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26090 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955591168 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 % _timestamp‚1.7885221555808933e+09 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚60690579 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚831291329910272 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 190451992 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11998444104704 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 208679560 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚930090962587648 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚842779550447104 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407872 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 208679558 trainer/global_step‚10099 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26090 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955591168 OIDeviceStatsMonitor.on_t€2ö#rain_batch_start/inactive_split.large_pool.current‚2 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 % _timestamp‚1.7885221555808933e+09 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚60690579 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚831291329910272 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 190451992 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11998444104704 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 208679560 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407872 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚842780467857920 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 253499582 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 226762752 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11998217341952 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 462153052 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11488238539776 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚824150050286667 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚812704636049616 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11446311648471 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411420 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _step‚829 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚842779550447104 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450780 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410816 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚831291311907328 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 129761415 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955591168 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚942089430632448 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214665 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 208679558 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1187 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 253473494 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686½g]µ4Ñ)/ _wandbruntime‚11782  _runtime‚117829*¡'Ò)ŒN OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 253524199 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 208699458 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚942179900177920 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899421184 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 208699458 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899421184 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚812782633183104 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 _step‚830 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 462197541 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11490250764288 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚630 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚824230053493101 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11489351343104 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 129773791 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11447420309997 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 462223657 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚831370960205824 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 253498085 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚812782616143744 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚942179643196928 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955591168 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚930180305748480 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚930180281808384 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11999361388544 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚824229139042113 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _runtime‚11782.242117638 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 208699456 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26114 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚60696988 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 253524199 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11446522898369 trainer/global_step‚10099 % _timestamp‚1.7885221567696905e+09 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 256980992 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 253498085 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11999594429440 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11489351343104 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177047040 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 129773789 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚60696360 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1187 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 401960960 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚831370960205824 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11490250764288 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 208699456 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚842861210970112 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 462223657 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31686 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 190470149 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007241 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚842861210970112 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 233040896 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411628 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 190470779 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚628 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728À3ƃ'Ó)‘N SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11999594429440 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11489351343104 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177047040 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 129773789 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚60696360 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1187 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 401960960 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚831370960205824 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11490250764288 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 208699456 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚842861210970112 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 462223657 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31686 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 190470149 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007241 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚842861210970112 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 233040896 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411628 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 190470779 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚628 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 253524199 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 208699458 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚942179900177920 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899421184 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 208699458 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899421184 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚812782633183104 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 _step‚830 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 462197541 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11490250764288 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚630 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚824230053493101 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11489351343104 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 129773791 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11447420309997 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 462223657 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚831370960205824 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 253498085 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚812782616143744 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚942179643196928 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955591168 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚930180305748480 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚930180281808384 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11999361388544 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚824229139042113 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _runtime‚11782.242117638 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 208699456 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26114 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚60696988 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 253524199 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11446522898369 trainer/global_step‚10099 % _timestamp‚1.7885221567696905e+09 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 256980992 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 253498085 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416¾:l'á4Ô)/ _wandbruntime‚11782  _runtime‚11782NLV`Õ)Ú 1loss/FBt1/assignment_loss‚0.38411701429867084 /loss/FRt4/detection_loss‚3.0944939756393435 .loss/FBt4/detection_loss‚3.031773547331492 %loss/total_loss‚36.84599338269943 epoch‚8 trainer/global_step‚10099 0loss/FRt1/assignment_loss‚2.2934207675918454 2loss/SRqqt1/assignment_loss‚1.0222374260305398 1loss/FBt2/assignment_loss‚0.11008572527264585 1loss/SRqqt3/detection_loss‚1.5074473863505586 /loss/FRt3/assignment_loss‚2.489240373053202 % _timestamp‚1.7885221567738907e+09 0loss/SRqqt4/detection_loss‚2.910172148987099 2loss/SRqqt4/assignment_loss‚1.5836511829660997 1loss/FBt4/assignment_loss‚0.23150993906650305 1loss/SRqqt2/detection_loss‚0.7905535324010998 /loss/FBt1/detection_loss‚1.9344099434939297 /loss/FBt2/detection_loss‚0.6061605363665672 _step‚831 0loss/FRt2/assignment_loss‚1.0489958776877477 .loss/FRt1/detection_loss‚1.831441110266643 /loss/FRt3/detection_loss‚2.4994572735414273 /loss/FBt3/detection_loss‚2.5324417515234514 /loss/FRt2/detection_loss‚1.5290491662991161 1loss/FBt3/assignment_loss‚0.17403310139392983 1loss/SRqqt1/detection_loss‚1.3463362953475406 _runtime‚11782.244140337 2loss/SRqqt3/assignment_loss‚1.0853121904561258 /loss/FRt4/assignment_loss‚2.046362419128418 2loss/SRqqt2/assignment_loss‚0.7632906982054313Œ ‘›eÖ)ß .loss/FBt4/detection_loss‚3.031773547331492 %loss/total_loss‚36.84599338269943 epoch‚8 trainer/global_step‚10099 0loss/FRt1/assignment_loss‚2.2934207675918454 2loss/SRqqt1/assignment_loss‚1.0222374260305398 1loss/FBt2/assignment_loss‚0.11008572527264585 1loss/SRqqt3/detection_loss‚1.5074473863505586 /loss/FRt3/assignment_loss‚2.489240373053202 % _timestamp‚1.7885221567738907e+09 0loss/SRqqt4/detection_loss‚2.910172148987099 2loss/SRqqt4/assignment_loss‚1.5836511829660997 1loss/FBt4/assignment_loss‚0.23150993906650305 1loss/SRqqt2/detection_loss‚0.7905535324010998 /loss/FBt1/detection_loss‚1.9344099434939297 /loss/FBt2/detection_loss‚0.6061605363665672 _step‚831 0loss/FRt2/assignment_loss‚1.0489958776877477 .loss/FRt1/detection_loss‚1.831441110266643 /loss/FRt3/detection_loss‚2.4994572735414273 /loss/FBt3/detection_loss‚2.5324417515234514 /loss/FRt2/detection_loss‚1.5290491662991161 1loss/FBt3/assignment_loss‚0.17403310139392983 1loss/SRqqt1/detection_loss‚1.3463362953475406 _runtime‚11782.244140337 2loss/SRqqt3/assignment_loss‚1.0853121904561258 /loss/FRt4/assignment_loss‚2.046362419128418 2loss/SRqqt2/assignment_loss‚0.7632906982054313 1loss/FBt1/assignment_loss‚0.38411701429867084 /loss/FRt4/detection_loss‚3.0944939756393435¿8‚¨“×)‚(:ˆ ´ÝêÔ„‰®€ gpu.0.gpu‚53.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚167.774' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚41.9435 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¿4#œâØ)‚(:Ô ´ÝêÔÖìÉ€$ disk./.usageGB‚2713.437587738037 disk.sda.in‚ 10.7578125 disk.sda2.in‚ 10.7578125 disk.sdb1.out‚0 disk.sdb2.out‚1348.712890625 disk.sda1.out‚0) disk./.usagePercent‚75.8ŸÊÔ?}6885061501316 disk.rbd0.out‚ 3.087890625 disk.nvme1n1.in‚ 498.80078125# disk.nvme1n1.out‚65520.55859375 disk.md0.out‚ 1300.1875$ memory_percent‚92.11604595184326 cpu‚25.212692752510147 network.recv‚ 3.806793e+06 disk.rbd0.in‚0 disk.sdb2.in‚ 7.0546875# disk.nvme0n1.out‚64946.67578125 disk.sda2.out‚1348.712890625 disk.rbd2.in‚0 disk.sda1.in‚0 disk.dm-0.out‚ 130458.484375 disk.sdb.in‚ 7.0546875 proc.cpu.threads‚17) proc.memory.availableMB‚ 1291.70703125 network.sent‚ 2.6543003e+07 disk.sda.out‚1348.712890625 disk.md0.in‚17.8125 disk.md0p1.out‚ 1300.1875 disk.sdb.out‚1348.712890625* proc.memory.percent‚18.320035934448242 disk.sdb1.in‚0 disk.dm-0.in‚ 1118.4375" proc.memory.rssMB‚ 3001.5546875 disk.rbd2.out‚3.25 disk.md0p1.in‚17.8125 disk.nvme0n1.in‚ 619.63671875îs4I’Ù)‚(:‡ ÃÝêÔúºëÿ gpu.0.gpu‚56.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚171.168' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.792 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚08å½"kÚ)‚(:à ÃÝêÔ¸ô€€ disk.rbd2.in‚0 disk.dm-0.in‚ 1118.4375 disk.md0p1.out‚ 1301.74609375 disk.rbd0.in‚0 disk.sdb1.out‚0" disk.nvme0n1.out‚ 65088.5078125 disk.sdb.in‚ 7.0546875 proc.cpu.threads‚17 network.recv‚ 3.81223e+06 disk.sda.out‚1350.3310546875! disk.sdb2.out‚1350.3310546875 disk.nvme0n1.in‚ 619.63671875% disk./.usageGB‚2713.6253356933594 disk.rbd2.out‚3.25 disk.sda1.out‚0! disk.dm-0.out‚130609.37109375 disk.sdb.out‚1350.3310546875" proc.memory.rssMB‚ 3001.5546875* proc.memory.percent‚18.320035934448242# memory_percent‚92.1177625656128 cpu‚25.212451116444534 disk.sda.in‚ 10.7578125 disk.sda2.in‚ 10.7578125 disk.md0.in‚17.8125 disk.md0.out‚ 1301.74609375) disk./.usagePercent‚75.87410012642259 disk.sdb1.in‚0 disk.sdb2.in‚ 7.0546875 network.sent‚ 2.6637768e+07 disk.nvme1n1.in‚ 498.80078125 disk.sda1.in‚0 disk.md0p1.in‚17.8125) proc.memory.availableMB‚ 1291.42578125! disk.sda2.out‚1350.3310546875 disk.rbd0.out‚ 3.087890625# disk.nvme1n1.out‚65520.86328125 «ö’Û)‚(:‡ ÒÝêÔ¶´¹ÿ gpu.0.gpu‚46.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚163.09' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚40.7725 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0%lšdÜ)‚(:Ù ÒÝêÔÄþ¥€ disk.sdb2.out‚1352.044921875 disk.sda1.in‚0* proc.memory.percent‚18.320035934448242% disk./.usageGB‚2713.6248207092285 disk.nvme0n1.in‚ 619.63671875" proc.memory.rssMB‚ 3001.5546875 proc.cpu.threads‚17 disk.sdb2.in‚ 7.0546875 disk.sda2.out‚1352.044921875 disk.md0.in‚17.8125 disk.rbd2.out‚3.25 network.recv‚ 3.816133e+06! disk.dm-0.out‚130837.37109375 disk.sdb.out‚1352.044921875 disk.rbd0.out‚ 3.087890625 disk.sdb1.out‚0 disk.md0p1.in‚17.8125 disk.rbd0.in‚0) proc.memory.availableMB‚ 1291.68359375 network.sent‚ 2.6645318e+07# disk.nvme0n1.out‚65316.30859375 disk.sda1.out‚0 disk.dm-0.in‚ 1118.4375 disk.sda.in‚ 10.7578125 disk.nvme1n1.in‚ 498.80078125 disk.md0.out‚ 1303.3984375 disk.rbd2.in‚0$ memory_percent‚92.11618900299072) disk./.usagePercent‚75.87408572725076 disk.md0p1.out‚ 1303.3984375 disk.sdb.in‚ 7.0546875 disk.sda.out‚1352.044921875 disk.sda2.in‚ 10.7578125 disk.nvme1n1.out‚ 65521.0625 cpu‚25.212218665095147 disk.sdb1.in‚0S+SÆ“Ý)‚(:ˆ áÝêÔôäÆÿ gpu.0.gpu‚30.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚165.266' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚41.3165 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¡ FlÞ)‚(:á áÝêÔ²€‚€ network.recv‚ 3.819677e+06" disk.sdb2.out‚1353.83349609375 cpu‚25.21199367775331 disk.sda1.out‚0 disk.dm-0.in‚ 1118.4375 network.sent‚ 2.6648414e+07) disk./.usagePercent‚75.88107583188389 disk.sda1.in‚0 disk.sda.in‚ 10.7578125" disk.sda2.out‚1353.83349609375 disk.md0.in‚17.8125( proc.memory.availableMB‚ 1291.6796875% disk./.usageGB‚2713.8748207092285 disk.md0p1.out‚ 1305.1328125 disk.nvme0n1.in‚ 619.63671875! disk.sda.out‚1353.83349609375 disk.rbd0.in‚0# disk.nvme1n1.out‚65521.35546875 disk.md0.out‚ 1305.1328125 disk.sdb2.in‚ 7.0546875! disk.sdb.out‚1353.83349609375 disk.sda2.in‚ 10.7578125* proc.memory.percent‚18.320035934448242 disk.sdb.in‚ 7.0546875 disk.nvme1n1.in‚ 498.80078125 disk.sdb1.out‚0$ memory_percent‚92.11621284484863" proc.memory.rssMB‚ 3001.5546875 proc.cpu.threads‚17 disk.rbd2.out‚3.25 disk.md0p1.in‚17.8125 disk.rbd0.out‚ 3.087890625 disk.rbd2.in‚0! disk.dm-0.out‚131123.93359375! disk.nvme0n1.out‚ 65610.828125 disk.sdb1.in‚00aúT4ß)/ _wandbruntime‚11839  _runtime‚11839Ès€ïÉà)à % _timestamp‚1.7885222137446432e+09 _runtime‚11839.215128957 _step‚832 & lr-AdamW/pg1‚6.559833199925056e-05 & lr-AdamW/pg2‚6.559833199925056e-05 trainer/global_step‚10149DasÎá)È % _timestamp‚1.7885222137446432e+09 _runtime‚11839.215128957 _step‚832 & lr-AdamW/pg1‚6.559833199925056e-05 & lr-AdamW/pg2‚6.559833199925056e-05 trainer/global_step‚10149À½ÑƒÙ4â)/ _wandbruntime‚11839  _runtime‚11839p’³(ã)P LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 254732484 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1212 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚828105416100955 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚835272826684928 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚620 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007241 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11544792527872 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚60976346 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11544792527872 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 191357179 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚846817637215744 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 254706396 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 130380213 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 228859904 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12056075163648 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11501758305579 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407872 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407872 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 130380215 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 209674460 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 _runtime‚11839.216606284 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 254706396 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚835272826684928 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411420 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 209674458 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚846816719804928 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚835272844687872 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11543893120000 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 209674458 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 252800000 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚835272844687872 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955591168 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 401960960 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚934556947564544 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12056304023552 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚828104501650175 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26090 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 464380854 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26088 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 254732484 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚846816719804928 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410816 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚618 trainer/global_step‚10149 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11543893120000 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26088 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚60976964 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0Åzîà (ä)†P R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚846816719804928 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410816 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚618 trainer/global_step‚10149 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11543893120000 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26088 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚60976964 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 254732484 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1212 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚828105416100955 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚835272826684928 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚620 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007241 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11544792527872 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚60976346 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11544792527872 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 191357179 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚846817637215744 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 254706396 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 130380213 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 228859904 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12056075163648 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11501758305579 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407872 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407872 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 130380215 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 209674460 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 _runtime‚11839.216606284 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 254706396 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚835272826684928 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411420 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 209674458 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚846816719804928 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚835272844687872 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11543893120000 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 209674458 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 252800000 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚835272844687872 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955591168 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 401960960 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚934556947564544 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12056304023552 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚828104501650175 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26090 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 464380854 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26088 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 254732484ÁL›Ä+4å)/ _wandbruntime‚11840  _runtime‚11840Bd½âæ)ŒN WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 130392591 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450988 % _timestamp‚1.7885222149511745e+09 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11502866967105 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955591168 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚846897462903808 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚934646242845184 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26114 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 248592384 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11501969555477 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 254757101 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚846898380327936 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 464451459 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚614 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 224652288 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11545005923328 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 130392589 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚835352474983424 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11545005923328 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917424128 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 254730987 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚946703723472384 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11545905344512 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚846898380327936 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 trainer/global_step‚10149 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12057232034816 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚828183590405621 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11545905344512 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 464425343 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 209694358 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411628 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007241 H@DeviceStatsMonitor.on_train_batch_end/segment.K@î[0small_pool.current‚564 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚934646266785280 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 _step‚834 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 209694358 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚835352456980480 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 191374592 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177047040 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚946703474880000 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 254730987 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚60982615 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚612 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 464451459 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899421184 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 254757101 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1187 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚816681620850144 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 209694356 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 191375206 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚8353524749834248¡¿'ç)‘N ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11545905344512 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 464425343 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 209694358 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411628 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007241 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚934646266785280 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 _step‚834 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 209694358 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚835352456980480 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 191374592 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177047040 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚946703474880000 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 254730987 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚60982615 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚612 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 464451459 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899421184 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 254757101 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1187 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚816681620850144 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 209694356 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 191375206 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚835352474983424 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 130392591 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450988 % _timestamp‚1.7885222149511745e+09 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11502866967105 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955591168 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚846897462903808 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚934646242845184 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26114 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 248592384 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11501969555477 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 254757101 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚846898380327936 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 464451459 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚614 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 224652288 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11545005923328 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 130392589 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚835352474983424 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11545005923328 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917424128 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 254730987 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚946703723472384 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11545905344512 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚846898380327936 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 trainer/global_step‚10149 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12057232034816 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚828183590405621ÂÌ)¡˜4è)/ _wandbruntime‚11840  _runtime‚118407 £‹_é)Ù 1loss/SRqqt2/detection_loss‚0.9629528818894988 /loss/FBt1/detection_loss‚2.2955221980810165 % _timestamp‚1.7885222149542496e+09 1loss/SRqqt4/detection_loss‚2.4198044147984734 0loss/FRt1/assignment_loss‚2.5045727605515338 0loss/FRt3/assignment_loss‚1.9330259439620106 /loss/FRt4/assignment_loss‚2.576717436313629 2loss/SRqqt2/assignment_loss‚1.0490533461713272 2loss/SRqqt4/assignment_loss‚1.5072868535744732 /loss/FBt3/detection_loss‚3.3487215538819632 %loss/total_loss‚38.70182995020458 0loss/FRt2/assignment_loss‚1.6598513863303446 1loss/FBt1/assignment_loss‚0.27833283811790466 1loss/FBt2/assignment_loss‚0.08394608457765121 1loss/FBt3/assignment_loss‚0.16973503345256327 /loss/FRt2/detection_loss‚1.6070927850224754 /loss/FRt4/detection_loss‚3.0213880240917206 1loss/SRqqt1/detection_loss‚1.5071796090265765 /loss/SRqqt3/detection_loss‚1.61517733912915 epoch‚8 /loss/FBt4/detection_loss‚2.6580998471805026 /loss/FRt1/detection_loss‚2.0313530466640044 trainer/global_step‚10149 _runtime‚11840.424600486 _step‚835 1loss/SRqqt3/assignment_loss‚1.268959339056164 .loss/FRt3/detection_loss‚2.226362752050839 /loss/FBt2/detection_loss‚0.3527518218799837 1loss/SRqqt1/assignment_loss‚1.319482055569037 1loss/FBt4/assignment_loss‚0.30446059883174065„&•dê)Þ 1loss/SRqqt4/detection_loss‚2.4198044147984734 0loss/FRt1/assignment_loss‚2.5045727605515338 0loss/FRt3/assignment_loss‚1.9330259439620106 /loss/FRt4/assignment_loss‚2.576717436313629 2loss/SRqqt2/assignment_loss‚1.0490533461713272 2loss/SRqqt4/assignment_loss‚1.5072868535744732 /loss/FBt3/detection_loss‚3.3487215538819632 %loss/total_loss‚38.70182995020458 0loss/FRt2/assignment_loss‚1.6598513863303446 1loss/FBt1/assignment_loss‚0.27833283811790466 1loss/FBt2/assignment_loss‚0.08394608457765121 1loss/FBt3/assignment_loss‚0.16973503345256327 /loss/FRt2/detection_loss‚1.6070927850224754 /loss/FRt4/detection_loss‚3.0213880240917206 1loss/SRqqt1/detection_loss‚1.5071796090265765 /loss/SRqqt3/detection_loss‚1.61517733912915 epoch‚8 /loss/FBt4/detection_loss‚2.6580998471805026 /loss/FRt1/detection_loss‚2.0313530466640044 trainer/global_step‚10149 _runtime‚11840.424600486 _step‚835 1loss/SRqqt3/assignment_loss‚1.268959339056164 .loss/FRt3/detection_loss‚2.226362752050839 /loss/FBt2/detection_loss‚0.3527518218799837 1loss/SRqqt1/assignment_loss‚1.319482055569037 1loss/FBt4/assignment_loss‚0.30446059883174065 1loss/SRqqt2/detection_loss‚0.9629528818894988 /loss/FBt1/detection_loss‚2.2955221980810165 % _timestamp‚1.7885222149542496e+09Ãû‰)¶“ë)‚(:ˆ ðÝêÔÁÈøÿ gpu.0.gpu‚47.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚37.0 gpu.0.powerWatts‚156.294' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚39.0735 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0CáÑjì)‚(:ß ðÝêÔ‚× disk.md0.out‚ 1306.69921875 disk.dm-0.out‚131333.7578125 disk.rbd0.in‚0 disk.sdb.in‚ 7.0546875 disk.nvme1n1.in‚ 498.80078125' proc.memory.availableMB‚ 1291.640625 network.sent‚ 2.6654551e+07 disk.sdb.out‚1355.466796875 disk.rbd0.out‚ 3.103515625" disk.nvme1n1.out‚ 65521.5546875 disk.rbd2.in‚0 disk.nvme0n1.in‚ 619.63671875 disk.md0.in‚17.8125! disk.nvme0n1.out‚ 65812.203125 disk.sda.in‚ 10.7578125 disk.md0p1.out‚ 1306.69921875 disk.sdb1.in‚0 disk.sdb2.out‚1355.466796875 disk.dm-0.in‚ 1118.4375 network.recv‚ 3.824247e+06) disk./.usagePercent‚75.88107583188389% disk./.usageGB‚2713.8748207092285 disk.sda.out‚1355.466796875 disk.sda2.out‚1355.466796875 disk.rbd2.out‚ 3.26171875 disk.sda1.out‚0$ memory_percent‚92.11645126342773" proc.memory.rssMB‚ 3001.5546875 cpu‚25.211761306243634 disk.sda2.in‚ 10.7578125 disk.sdb2.in‚ 7.0546875 disk.sda1.in‚0 disk.md0p1.in‚17.8125 proc.cpu.threads‚17 disk.sdb1.out‚0* proc.memory.percent‚18.320035934448242 ?m’í)‚(:‡ ÿÝêÔ±×ÿ gpu.0.gpu‚56.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚37.0 gpu.0.powerWatts‚132.732' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚33.183 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ºÔIÄfî)‚(:Û ÿÝêÔ‰õñÿ disk.sdb1.in‚0 disk.rbd0.in‚0 disk.md0.in‚17.8125 proc.cpu.threads‚17 disk.sda.out‚ 1356.94921875 disk.sdb1.out‚0$ memory_percent‚92.11645126342773" proc.memory.rssMB‚ 3001.5546875* proc.memory.percent‚18.320035934448242 cpu‚25.211522421791486 disk.nvme0n1.in‚ 619.63671875 disk.nvme1n1.out‚ 65521.84375 disk.dm-0.in‚ 1118.4375! disk.dm-0.out‚131759.49609375 network.recv‚ 3.829517e+06 disk.md0p1.out‚ 1308.12109375% disk./.usageGB‚2713.8748626708984 disk.nvme1n1.in‚ 498.80078125 disk.sdb2.in‚ 7.0546875# disk.nvme0n1.out‚66237.65234375 disk.md0.out‚ 1308.12109375 disk.rbd2.in‚0 disk.sdb2.out‚ 1356.94921875 disk.sda1.in‚0 disk.md0p1.in‚17.8125' proc.memory.availableMB‚ 1291.640625 network.sent‚ 2.676428e+07) disk./.usagePercent‚75.88107700514975 disk.sda.in‚ 10.7578125 disk.sda2.in‚ 10.7578125 disk.sda2.out‚ 1356.94921875 disk.sdb.out‚ 1356.94921875 disk.rbd0.out‚ 3.166015625 disk.rbd2.out‚ 3.26171875 disk.sda1.out‚0 disk.sdb.in‚ 7.0546875Ž„î’ï)‚(:‡ ŽÞêÔãðÿ gpu.0.gpu‚38.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚168.992' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.248 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0FºÚMnð)‚(:ã ŽÞêÔïöŠ€ network.recv‚ 3.834129e+06 disk.md0p1.out‚ 1309.6953125 disk.nvme0n1.out‚ 66237.65625 disk.nvme1n1.in‚ 498.80078125 proc.cpu.threads‚17 disk.md0p1.in‚17.8125 disk.sda2.in‚ 10.7578125! disk.nvme1n1.out‚ 65522.046875! disk.sda.out‚1358.58447265625$ memory_percent‚92.11647510528564" disk.sda2.out‚1358.58447265625) disk./.usagePercent‚75.88107711181027 disk.sdb.in‚ 7.0546875 disk.sda.in‚ 10.7578125 disk.md0.out‚ 1309.6953125 disk.sdb2.in‚ 7.0546875 disk.sda1.in‚0 disk.sda1.out‚0) proc.memory.availableMB‚ 1291.63671875 disk.sdb1.in‚0 disk.md0.in‚17.8125 disk.rbd2.in‚0 disk.rbd2.out‚ 3.32421875 disk.dm-0.out‚ 131759.703125 cpu‚25.21129704715124 network.sent‚ 2.6770598e+07% disk./.usageGB‚2713.8748664855957! disk.sdb.out‚1358.58447265625 disk.rbd0.in‚0 disk.rbd0.out‚ 3.166015625 disk.dm-0.in‚ 1118.4375* proc.memory.percent‚18.320035934448242 disk.nvme0n1.in‚ 619.63671875 disk.sdb1.out‚0" disk.sdb2.out‚1358.58447265625" proc.memory.rssMB‚ 3001.5546875Ö{‡pñ)‚(:å ÞêÔ‰’üþ% disk./.usageGB‚2713.3447647094727 disk.rbd2.out‚ 3.32421875 disk.md0p1.in‚17.8125 disk.nvme0n1.in‚ 619.63671875 disk.sda2.in‚ 10.7578125 network.sent‚ 2.6775855e+07 disk.sda1.in‚0 disk.rbd0.in‚0# disk.nvme0n1.out‚66438.37109375! disk.sda.out‚1360.59228515625" disk.sda2.out‚1360.59228515625) disk./.usagePercent‚75.86625524428504 disk.md0.in‚17.8125" proc.memory.rssMB‚ 3001.5546875 cpu‚25.211078034233186 disk.nvme1n1.in‚ 498.80078125 disk.sdb2.in‚ 7.0546875 disk.md0p1.out‚ 1311.640625 disk.sda.in‚ 10.7578125 disk.md0.out‚ 1311.640625" disk.sdb2.out‚1360.59228515625 disk.dm-0.in‚ 1118.4375* proc.memory.percent‚18.320035934448242 disk.nvme1n1.out‚ 65522.3125 disk.sda1.out‚0 disk.sdb.in‚ 7.0546875! disk.sdb.out‚1360.59228515625 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.rbd2.in‚0! disk.dm-0.out‚131960.68359375 disk.rbd0.out‚ 3.25341796875$ memory_percent‚92.11492538452148' proc.memory.availableMB‚ 1291.890625 proc.cpu.threads‚17 network.recv‚ 3.837696e+06»Œäœ”ò)‚(:‰ ÞêÔ¤†Íÿ gpu.0.gpu‚55.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚170.219' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.55475 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0hÂ_4ó)/ _wandbruntime‚11897  _runtime‚11897°ü=hÇô)Á % lr-AdamW/pg2‚6.23455983771229e-05 trainer/global_step‚10199 % _timestamp‚1.7885222719589305e+09 _runtime‚11897.430837274 _step‚836 % lr-AdamW/pg1‚6.23455983771229e-05%'ŸÌõ)Æ % lr-AdamW/pg1‚6.23455983771229e-05 % lr-AdamW/pg2‚6.23455983771229e-05 trainer/global_step‚10199 % _timestamp‚1.7885222719589305e+09 _runtime‚11897.430837274 _step‚836Ä54å 4ö)/ _wandbruntime‚11897  _runtime‚11897Ê9ÈP$÷)‚P M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 255939298 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚850853889162752 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 255939298 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407872 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚939022908601344 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 210669360 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚820502662501776 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚832059867464463 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214665 _runtime‚11897.431357687 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 401960960 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚832058953013683 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚850854806573568 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177047040 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚820502645462416 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1187 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 130999013 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 255965386 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26088 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955591168 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚839254341462528 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚951137240679936 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26090 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11557204962687 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚850854806573568 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 466634746 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 192261740 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11600447108096 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 255965386 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚634 trainer/global_step‚10199 JDevice¦-Üb¸StatsMonitor.on_train_batch_start/allocation.all.current‚26090 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26088 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12114096927232 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚22{Y (ø)‡P RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 192262376 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚61263361 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411420 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11599547700224 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚61262727 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450780 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11556307551267 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955591168 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚839254341462528 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚951137240679936 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26090 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11557204962687 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚850854806573568 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 466634746 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 192261740 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11600447108096 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 255965386 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚634 trainer/global_step‚10199 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26088 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12114096927232 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 255939298 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚850853889162752 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 255939298 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407872 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚939022908601344 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 210669360 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚820502662501776 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚832059867464463 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214665 _runtime‚11897.431357687 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 401960960 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚832058953013683 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚850854806573568 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177047040 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚820502645462416 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1187 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 130999013 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 255965386 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26088 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 466679261 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450988 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11558313624213 _step‚838 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚850935549685760 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 131011389 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚939112203881984 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 466653145 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26114 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚832138956220117 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚820580642595904 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411628 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚10199 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚839333989761024 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚850934632261632 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚839333989761024 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 131011391 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚641 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11601559924736 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚951227712345088 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚939112227822080 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11557416212585 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007241 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚643 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 192279940 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12115484523008 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11600660503552 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177047040 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 _runtime‚11898.618525952 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚839333971758080 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899421184 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26114 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 210689256 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 210689256 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 255990003 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917424128 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 401960960%.'û)‘N T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚850935549685760 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11601559924736 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚61269192 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚61268551 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 % _timestamp‚1.7885222731461816e+09 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 210689258 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31686 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚939112227822080 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11557416212585 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007241 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚643 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 192279940 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12115484523008 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11600660503552 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177047040 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 _runtime‚11898.618525952 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚839333971758080 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899421184 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26114 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 210689256 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 210689256 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 255990003 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917424128 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 401960960 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 255963889 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177047040 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899421184 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚839333971758080 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 252786688 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚832138041769129 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 466679261 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11600660503552 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚951227459558400 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 466679261 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450988 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11558313624213 _step‚838 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚850935549685760 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 131011389 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚939112203881984 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 466653145 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26114 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚832138956220117 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚820580642595904 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411628 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚10199 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚839333989761024 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚850934632261632 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚839333989761024 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 131011391 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚641 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11601559924736 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚951227712345088 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 BiΟ”*‚(:‰ »ÞêÔ©¢Ÿÿ gpu.0.gpu‚44.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚167.157' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚41.78925 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚075iOr‚*‚(:ç »ÞêÔ·§òÿ disk.sdb.in‚ 7.0546875 disk.nvme1n1.in‚ 498.80078125 disk.sdb1.in‚0 disk.dm-0.in‚ 1118.4375! disk.dm-0.out‚132349.30859375 disk.md0p1.out‚ 1314.546875! disk.sdb.out‚1363.60498046875 disk.sda.in‚ 10.7578125 network.sent‚ 2.6889777e+07 disk.rbd2.in‚0 disk.sdb2.in‚ 7.0546875 disk.sda1.in‚0 disk.sda1.out‚0! disk.nvme0n1.out‚ 66826.515625 network.recv‚ 3.848172e+06 disk.nvme0n1.in‚ 619.63671875" disk.sda2.out‚1363.60498046875$ memory_percent‚92.11688041687012( proc.memory.availableMB‚ 1291.5703125" proc.memory.rssMB‚ 3001.5546875* proc.memory.percent‚18.320035934448242 cpu‚25.210606845840505 disk.md0.in‚17.8125 disk.sda2.in‚ 10.7578125% disk./.usageGB‚2712.3178901672363# disk.nvme1n1.out‚65522.79296875 disk.sdb1.out‚0 disk.rbd2.out‚ 3.4033203125 proc.cpu.threads‚17) disk./.usagePercent‚75.83754340230374 disk.rbd0.out‚ 3.31640625" disk.sdb2.out‚1363.60498046875 disk.md0p1.in‚17.8125! disk.sda.out‚1363.60498046875 disk.rbd0.in‚0 disk.md0.out‚ 1314.546875ð¹³”ƒ*‚(:‰ ÊÞêÔ˜‰Ùÿ gpu.0.gpu‚44.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚160.637' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚40.15925 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0êýBpr„*‚(:ç ÊÞêÔˆ–™€ disk.sdb1.out‚0 disk.md0.out‚ 1316.49609375 disk.sdb.in‚ 7.0546875 disk.rbd0.out‚ 3.31640625$ disk./.usageGB‚2712.320770263672 disk.rbd2.in‚0 disk.md0p1.in‚17.8125 disk.nvme1n1.in‚ 498.80078125 network.sent‚ 2.6893852e+07 disk.sdb2.in‚ 7.0546875 network.recv‚ 3.852816e+06 disk.md0.in‚17.8125" disk.sdb2.out‚1365.60498046875 disk.md0p1.out‚ 1316.49609375' proc.memory.availableMB‚ 1292.015625 disk.sdb1.in‚0 disk.sda1.out‚0! disk.nvme0n1.out‚ 66849.390625! disk.sdb.out‚1365.60498046875 disk.nvme0n1.in‚ 619.63671875 disk.sda.in‚ 10.7578125! disk.sda.out‚1365.60498046875" disk.sda2.out‚1365.60498046875" disk.nvme1n1.out‚ 65522.9921875 cpu‚25.21038329814644 disk.rbd2.out‚ 3.46630859375 disk.sda1.in‚0 disk.dm-0.out‚132372.3828125 proc.cpu.threads‚17) disk./.usagePercent‚75.83762393100548 disk.dm-0.in‚ 1118.4375 disk.sda2.in‚ 10.7578125 disk.rbd0.in‚0$ memory_percent‚92.11416244506836" proc.memory.rssMB‚ 3001.5546875* proc.memory.percent‚18.320035934448242«›Y4…*/ _wandbruntime‚11949  _runtime‚11949{¾ûdž*Á % lr-AdamW/pg2‚5.91692437855254e-05 trainer/global_step‚10249 % _timestamp‚1.7885223244656024e+09 _runtime‚11949.935959153 _step‚840 % lr-AdamW/pg1‚5.91692437855254e-05ä(Ìṡ*Æ % _timestamp‚1.7885223244656024e+09 _runtime‚11949.935959153 _step‚840 % lr-AdamW/pg1‚5.91692437855254e-05 % lr-AdamW/pg2‚5.91692437855254e-05 trainer/global_step‚10249È+³þê4ˆ*/ _wandbruntime‚11949  _runtime‚11949ÌqŸ(‰*‚P ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚836013404377191 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 trainer/global_step‚10249 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917410816 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚824401667208176 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12172371973632 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚854891058520576 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 257198288 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955591168 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410816 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1187 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 401960960 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12172145210880 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚824401650168816 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411420 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚955661241611776 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407872 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 250702848 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 193168627 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 131617813 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚943488869638144 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26088 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26088 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 257172200 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11611754208375 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚843235874243072 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 211664258 % _timestamp‚1.7885223244665973e+09 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177047040 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚61551454 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214665 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 468836458 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 211664260 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 468862548 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 193169269 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11655202280448 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26090 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚836014318827971 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955591168 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11656101688320 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 211664260 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚943488845698048 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚854891975931392 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚843235874243072 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚843235856240128 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0Õ]e (Š*‡P YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚854891975931392 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 211664258 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11655202280448 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚640 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 211664260 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚943488845698048 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚854891975931392 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚843235874243072 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚843235856240128 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚836013404377191 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 trainer/global_step‚10249 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917410816 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚824401667208176 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12172371973632 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚854891058520576 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 257198288 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955591168 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410816 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1187 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 401960960 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12172145210880 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚824401650168816 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411420 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚955661241611776 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407872 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 250702848 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 193168627 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 131617813 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚943488869638144 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26088 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26088 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 257172200 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11611754208375 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚843235874243072 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 211664258 % _timestamp‚1.7885223244665973e+09 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177047040 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚61551454 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214665 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 468836458 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 211664260 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 468862548 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 193169269 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11655202280448 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26090 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚836014318827971 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955591168 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11656101688320 J8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11656315083776 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚61556460 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 257222905 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 211684156 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31686 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11657214504960 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚854971801619456 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 468880947 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚943578188858880 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚824479630262944 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚843315486535680 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 131630191 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 211684158 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 257196791 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚632 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 _runtime‚11950.876859993 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177047040 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚843315486535680 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 259078144 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚630 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚955751458264064 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007241 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 trainer/global_step‚10249 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11612862869693 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11657214504960 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚842 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 468907063 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899421184 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚955751717342208 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 401960960 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411628 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 211684158 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 211684156 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚943578164918784 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 131630189 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚843315504538624 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1187 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚836093407583625 % _timestamp‚1.7885223254058907e+09 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26114 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955591168 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899421184 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚824479647302304 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 468907063 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚61557090 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚854972719043584 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26114 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚854972719043584 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 468880947 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 257222905 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 257196791 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 193186649 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚843315504538624 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 235138048 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11613760281321 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917424128³:ï'*‘N J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31686 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11657214504960 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚854971801619456 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 468880947 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚943578188858880 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚824479630262944 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚843315486535680 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 131630191 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 211684158 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 257196791 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚632 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 _runtime‚11950.876859993 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177047040 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚843315486535680 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 259078144 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚630 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚955751458264064 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007241 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 trainer/global_step‚10249 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11612862869693 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11657214504960 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚842 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 468907063 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899421184 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚955751717342208 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 401960960 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411628 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 211684158 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 211684156 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚943578164918784 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 131630189 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚843315504538624 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1187 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚836093407583625 % _timestamp‚1.7885223254058907e+09 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26114 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955591168 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899421184 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚824479647302304 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 468907063 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚61557090 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚854972719043584 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26114 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚854972719043584 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 468880947 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 257222905 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 257196791 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 193186649 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚843315504538624 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 235138048 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11613760281321 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917424128 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177047040 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11656315083776 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚61556460 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 257222905 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 211684156 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728ÊO÷,ö4Ž*/ _wandbruntime‚11950  _runtime‚11950ÂAç]*× 1loss/FBt2/assignment_loss‚0.05318006091509388 % _timestamp‚1.7885223254082847e+09 1loss/SRqqt1/detection_loss‚1.3606714248933174 1loss/SRqqt4/detection_loss‚2.3490069336616077 .loss/FBt3/detection_loss‚2.065100872516632 /loss/FBt4/detection_loss‚2.2268557594372678 %loss/total_loss‚39.70945399867598 epoch‚8 /loss/FRt4/assignment_loss‚3.627442040864159 1loss/FBt1/assignment_loss‚0.17741061700016303 /loss/FBt1/detection_loss‚2.0667489655315876 .loss/FRt4/detection_loss‚3.667249967070187 trainer/global_step‚10249 2loss/SRqqt1/assignment_loss‚1.1423687126694455 1loss/FBt4/assignment_loss‚0.22778333180331872 /loss/FRt1/detection_loss‚1.9372135519415365 0loss/SRqqt2/detection_loss‚0.888057485545675 1loss/SRqqt3/detection_loss‚1.6539166366251616 _step‚843 0loss/FRt2/assignment_loss‚1.3115251739819844 1loss/SRqqt3/assignment_loss‚1.450617232431586 0loss/FBt3/assignment_loss‚0.5018534522037953 /loss/FRt3/detection_loss‚2.9717779480852187 _runtime‚11950.87840278 0loss/FRt1/assignment_loss‚2.4708247801353194 1loss/SRqqt2/assignment_loss‚1.032614872381091 2loss/SRqqt4/assignment_loss‚1.4898908135409539 /loss/FRt2/detection_loss‚1.3363394918851554 /loss/FBt2/detection_loss‚0.7018326742786772 0loss/FRt3/assignment_loss‚2.9991711992770433gË‚Fb*Ü trainer/global_step‚10249 2loss/SRqqt1/assignment_loss‚1.1423687126694455 1loss/FBt4/assignment_loss‚0.22778333180331872 /loss/FRt1/detection_loss‚1.9372135519415365 0loss/SRqqt2/detection_loss‚0.888057485545675 1loss/SRqqt3/detection_loss‚1.6539166366251616 _step‚843 0loss/FRt2/assignment_loss‚1.3115251739819844 1loss/SRqqt3/assignment_loss‚1.450617232431586 0loss/FBt3/assignment_loss‚0.5018534522037953 /loss/FRt3/detection_loss‚2.9717779480852187 _runtime‚11950.87840278 0loss/FRt1/assignment_loss‚2.4708247801353194 1loss/SRqqt2/assignment_loss‚1.032614872381091 2loss/SRqqt4/assignment_loss‚1.4898908135409539 /loss/FRt2/detection_loss‚1.3363394918851554 /loss/FBt2/detection_loss‚0.7018326742786772 0loss/FRt3/assignment_loss‚2.9991711992770433 1loss/FBt2/assignment_loss‚0.05318006091509388 % _timestamp‚1.7885223254082847e+09 1loss/SRqqt1/detection_loss‚1.3606714248933174 1loss/SRqqt4/detection_loss‚2.3490069336616077 .loss/FBt3/detection_loss‚2.065100872516632 /loss/FBt4/detection_loss‚2.2268557594372678 %loss/total_loss‚39.70945399867598 epoch‚8 /loss/FRt4/assignment_loss‚3.627442040864159 1loss/FBt1/assignment_loss‚0.17741061700016303 /loss/FBt1/detection_loss‚2.0667489655315876 .loss/FRt4/detection_loss‚3.667249967070187˺B’‘*‚(:‡ ÙÞêÔתǀ gpu.0.gpu‚57.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚37.0 gpu.0.powerWatts‚124.044' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚31.011 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ño¾Èv’*‚(:ë ÙÞêÔ¸±â€ disk.dm-0.in‚ 1118.4375 disk.md0p1.in‚17.8125 disk.md0p1.out‚ 1317.91015625" disk.nvme1n1.out‚ 65523.2265625 disk.md0.out‚ 1317.91015625# disk.nvme0n1.out‚66918.66796875$ disk./.usageGB‚2712.355094909668 disk.sda.out‚1367.0771484375! disk.sda2.out‚1367.0771484375 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.sdb.in‚ 7.0546875 disk.rbd0.in‚0! disk.dm-0.out‚132441.89453125 disk.nvme0n1.in‚ 619.63671875 cpu‚25.21013821145068 disk.nvme1n1.in‚ 498.80078125 disk.md0.in‚17.8125 disk.rbd2.out‚ 3.46630859375 disk.sda1.in‚0 network.recv‚ 3.856737e+06 disk.rbd0.out‚ 3.37939453125 disk.rbd2.in‚0 disk.sdb2.in‚ 7.0546875$ memory_percent‚92.11595058441162) proc.memory.availableMB‚ 1291.72265625" proc.memory.rssMB‚ 3001.5546875 disk.sda2.in‚ 10.7578125 disk.sda1.out‚0* proc.memory.percent‚18.320035934448242 proc.cpu.threads‚17 network.sent‚ 2.6899457e+07) disk./.usagePercent‚75.83858366247351 disk.sdb.out‚1367.0771484375 disk.sda.in‚ 10.7578125! disk.sdb2.out‚1367.0771484375-•P2’“*‚(:‡ èÞêÔÉÑþ gpu.0.gpu‚51.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚36.0 gpu.0.powerWatts‚121.876' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚30.469 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0²tVh”*‚(:Ý èÞêÔߊÎÿ" proc.memory.rssMB‚ 3001.5546875 network.recv‚ 3.862344e+06$ disk./.usageGB‚2712.620918273926 disk.sda2.in‚ 10.7578125 disk.md0.in‚17.8125 disk.dm-0.in‚ 1118.4375$ memory_percent‚92.11587905883789 cpu‚25.209818038407594 disk.nvme1n1.in‚ 498.80078125 disk.rbd2.in‚0 disk.nvme0n1.in‚ 619.63671875 disk.rbd0.out‚ 3.37939453125 disk.sdb1.out‚0) disk./.usagePercent‚75.84601619499388 disk.rbd2.out‚ 3.529296875! disk.sdb2.out‚1368.5283203125 disk.sda1.in‚0 disk.sda1.out‚0 proc.cpu.threads‚17" disk.nvme0n1.out‚ 67066.4765625 disk.sdb.in‚ 7.0546875 disk.sdb1.in‚0 disk.sda.in‚ 10.7578125 disk.dm-0.out‚ 132589.875' proc.memory.availableMB‚ 1291.734375 network.sent‚ 2.700927e+07 disk.sda.out‚1368.5283203125! disk.sda2.out‚1368.5283203125 disk.rbd0.in‚0 disk.sdb2.in‚ 7.0546875* proc.memory.percent‚18.320035934448242" disk.nvme1n1.out‚ 65523.3984375 disk.md0.out‚ 1319.296875 disk.md0p1.in‚17.8125 disk.md0p1.out‚ 1319.296875 disk.sdb.out‚1368.5283203125ª}+v’•*‚(:‡ ÷ÞêÔæ­–ÿ gpu.0.gpu‚56.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚37.0 gpu.0.powerWatts‚114.408' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.602 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ŽyD.r–*‚(:ç ÷ÞêÔ³…¿ÿ# disk.nvme1n1.out‚65523.67578125 disk.rbd2.out‚ 3.529296875 disk.sda1.in‚0 disk.md0p1.out‚ 1320.78515625$ memory_percent‚92.11592674255371( proc.memory.availableMB‚ 1291.7265625 disk.sdb.out‚1370.0810546875 disk.sdb1.out‚0 disk.md0.in‚17.8125 disk.md0.out‚ 1320.78515625! disk.sdb2.out‚1370.0810546875 network.recv‚ 3.866756e+06 disk.nvme0n1.in‚ 619.63671875# disk.nvme0n1.out‚67286.87890625 disk.nvme1n1.in‚ 498.80078125 cpu‚25.209452407984067 proc.cpu.threads‚17 disk.rbd0.out‚ 3.37939453125 disk.sdb1.in‚0* proc.memory.percent‚18.320035934448242! disk.sda2.out‚1370.0810546875 disk.sda1.out‚0" proc.memory.rssMB‚ 3001.5546875 disk.dm-0.in‚ 1118.4375 disk.md0p1.in‚17.8125$ disk./.usageGB‚2712.874897003174 disk.sdb.in‚ 7.0546875 disk.sda.out‚1370.0810546875 disk.rbd0.in‚0 disk.sdb2.in‚ 7.0546875 disk.dm-0.out‚132810.5546875 network.sent‚ 2.7013551e+07' disk./.usagePercent‚75.853117546562 disk.sda.in‚ 10.7578125 disk.sda2.in‚ 10.7578125 disk.rbd2.in‚0Ç;Àm“—*‚(:ˆ †ßêÔþ£ÿ gpu.0.gpu‚54.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚39.0 gpu.0.powerWatts‚168.822' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.2055 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚04]­s˜*‚(:è †ßêÔô¿¼€ disk.md0p1.out‚ 1322.8046875" disk.sdb2.out‚1372.15966796875 disk.sda1.in‚0 disk.sda.in‚ 10.7578125* proc.memory.percent‚18.320035934448242! disk.sda.out‚1372.15966796875 disk.sda2.in‚ 10.7578125 disk.md0.in‚17.8125 network.recv‚ 3.871173e+06$ disk./.usageGB‚2712.874897003174 disk.dm-0.in‚ 1118.4375 disk.rbd0.in‚0$ memory_percent‚92.11742877960205 disk.rbd2.out‚ 3.529296875 disk.md0p1.in‚17.8125 disk.md0.out‚ 1322.8046875 cpu‚25.20914534023236 network.sent‚ 2.7019871e+07! disk.sdb.out‚1372.15966796875 disk.nvme1n1.in‚ 498.80078125" proc.memory.rssMB‚ 3001.5546875' disk./.usagePercent‚75.853117546562 disk.sda1.out‚0" disk.sda2.out‚1372.15966796875 disk.sdb1.in‚0 disk.sdb1.out‚0) proc.memory.availableMB‚ 1291.48046875# disk.nvme1n1.out‚65523.87890625 disk.rbd2.in‚0! disk.dm-0.out‚132810.76953125! disk.nvme0n1.out‚ 67286.890625 disk.sdb.in‚ 7.0546875 proc.cpu.threads‚17 disk.sdb2.in‚ 7.0546875 disk.nvme0n1.in‚ 619.63671875 disk.rbd0.out‚ 3.37939453125áÑw4™*/ _wandbruntime‚12002  _runtime‚12002eK1Ëš*Å trainer/global_step‚10299 % _timestamp‚1.7885223768261547e+09 _runtime‚12002.296969336 _step‚844 ' lr-AdamW/pg1‚5.6069930171994125e-05 ' lr-AdamW/pg2‚5.6069930171994125e-05aÝéÞЛ*Ê ' lr-AdamW/pg2‚5.6069930171994125e-05 trainer/global_step‚10299 % _timestamp‚1.7885223768261547e+09 _runtime‚12002.296969336 _step‚844 ' lr-AdamW/pg1‚5.6069930171994125e-05̼àPN4œ*/ _wandbruntime‚12002  _runtime‚12002 ßU"*P F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 471064260 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917410816 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407872 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚640 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26090 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 212659160 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11710856860672 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚858928227878400 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 132236613 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11668098276903 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410816 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚642 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 212659160 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚61839603 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚839967855740699 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 $ _timestamp‚1.788522376827094e+09 _runtime‚12002.298454443 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1187 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26088 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214665 trainer/global_step‚10299 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 401960960 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11667200865483 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007241 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 471090350 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1212 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11710856860672 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26088 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚960185381933056 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955591168 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚847217389020672 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚839968770191479 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 194076858 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚947954806734848 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12230551258112 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 212659158 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚960185131230208 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 258405102 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407872 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 258431190 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚61840243 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚858929145289216 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 132236615 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 250702848 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚858928227878400 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11711756268544 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 212659158 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚947954830674944 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 226762752 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12230324495360 JDDeviceStatsMonitor.on_train_batch_starL>²t/oversize_allocations.current‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚858929145289216 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 258431190 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 _step‚845 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚828300654875216 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚847217371017728 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚847217371017728 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11711756268544 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607L ™4 (ž*†P QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410816 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚642 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 212659160 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚61839603 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚839967855740699 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 $ _timestamp‚1.788522376827094e+09 _runtime‚12002.298454443 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1187 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26088 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214665 trainer/global_step‚10299 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 401960960 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11667200865483 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007241 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 471090350 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1212 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11710856860672 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26088 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚960185381933056 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955591168 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚847217389020672 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚839968770191479 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 194076858 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚947954806734848 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12230551258112 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 212659158 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚960185131230208 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 258405102 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407872 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 258431190 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚61840243 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚858929145289216 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 132236615 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 250702848 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚858928227878400 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11711756268544 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 212659158 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚947954830674944 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 226762752 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12230324495360 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚858929145289216 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 258431190 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 _step‚845 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚828300654875216 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚847217371017728 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚847217371017728 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11711756268544 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 471064260 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917410816 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407872 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚640 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26090 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 212659160 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11710856860672 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚858928227878400 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 132236613 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11668098276903 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944Í@A¿#4Ÿ*/ _wandbruntime‚12003  _runtime‚12003‡Ù·B' *ŒN ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚847297001313280 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450988 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 212679056 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 471134865 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955591168 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 194094401 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 258455807 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11711969664000 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 258429693 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899421184 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 471134865 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚61845412 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411628 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007241 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 trainer/global_step‚10299 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 471108749 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 222555136 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 471108749 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 132248991 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 212679058 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚847297019316224 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26114 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚859008970977280 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚626 _runtime‚12003.276698343 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11711969664000 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚61846038 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11712869085184 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚840047858947133 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚828378652008704 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11712869085184 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12231715849216 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚828378634969344 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚847297019316224 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26114 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚847297001313280 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1187 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚948044149895680 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31686 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 258455807 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 212679056 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 194094401 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 258455807 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11711969664000 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 258429693 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899421184 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 471134865 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚61845412 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411628 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007241 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 trainer/global_step‚10299 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 471108749 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 222555136 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 471108749 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 132248991 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 212679058 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚847297019316224 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26114 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚859008970977280 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚626 _runtime‚12003.276698343 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11711969664000 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚61846038 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11712869085184 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚840047858947133 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚828378652008704 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11712869085184 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12231715849216 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚828378634969344 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚847297019316224 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26114 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚847297001313280 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1187 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚948044149895680 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31686 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 258455807 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 212679056 F;”§*‚(:‰ ¤ßêԥǪþ gpu.0.gpu‚47.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚37.0 gpu.0.powerWatts‚115.025' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.75625 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0æ…¹àj¨*‚(:ß ¤ßêÔï΀€ disk.sdb2.in‚ 7.0546875 disk.sda1.out‚0! disk.sda2.out‚1375.5615234375( disk./.usagePercent‚75.8670961559203 disk.nvme0n1.in‚ 619.63671875 disk.rbd2.in‚0! disk.sdb2.out‚1375.5615234375$ disk./.usageGB‚2713.374839782715 disk.nvme1n1.in‚ 498.80078125 disk.md0.out‚ 1326.09375 cpu‚25.208647665176407# disk.nvme0n1.out‚67739.16796875 disk.rbd0.in‚0 disk.rbd2.out‚ 3.529296875 disk.sda1.in‚0 disk.dm-0.in‚ 1118.4375* proc.memory.percent‚18.320035934448242 disk.md0p1.in‚17.8125 disk.sdb.out‚1375.5615234375# disk.nvme1n1.out‚65524.25390625 disk.md0.in‚17.8125 disk.dm-0.out‚ 133263.421875& proc.memory.availableMB‚ 1291.90625" proc.memory.rssMB‚ 3001.5546875 network.sent‚ 2.7132829e+07 network.recv‚ 3.880241e+06 disk.sdb.in‚ 7.0546875 disk.sda2.in‚ 10.7578125 disk.md0p1.out‚ 1326.09375 disk.sda.in‚ 10.7578125 disk.sda.out‚1375.5615234375$ memory_percent‚92.11483001708984 proc.cpu.threads‚17 disk.rbd0.out‚ 3.37939453125 disk.sdb1.in‚0 disk.sdb1.out‚0‹Þ»˜’©*‚(:‡ ³ßêÔÇòŠþ gpu.0.gpu‚54.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚159.696' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚39.924 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Uqipª*‚(:å ³ßêÔ‰¯îÿ disk.md0p1.in‚17.8125 disk.sda.out‚1376.9501953125 disk.sda2.in‚ 10.7578125# disk.nvme1n1.out‚65524.69921875 cpu‚25.20836819664023) disk./.usagePercent‚75.85226362234185# disk.nvme0n1.out‚68049.54296875 disk.sdb.in‚ 7.0546875 disk.sdb.out‚1376.9501953125 disk.nvme1n1.in‚ 498.80078125 disk.rbd2.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 network.recv‚ 3.884278e+06% disk./.usageGB‚2712.8443565368652 disk.md0.in‚17.8125 disk.rbd2.out‚ 3.529296875 disk.dm-0.in‚ 1118.4375* proc.memory.percent‚18.320035934448242 proc.cpu.threads‚17 disk.sda.in‚ 10.7578125 disk.rbd0.out‚ 3.37939453125$ memory_percent‚92.11664199829102 disk.rbd0.in‚0! disk.sdb2.out‚1376.9501953125" proc.memory.rssMB‚ 3001.5546875 network.sent‚ 2.714108e+07 disk.nvme0n1.in‚ 619.63671875 disk.sdb1.out‚0 disk.dm-0.out‚133574.2421875' proc.memory.availableMB‚ 1291.609375 disk.md0p1.out‚ 1327.4140625! disk.sda2.out‚1376.9501953125 disk.sdb1.in‚0 disk.md0.out‚ 1327.4140625 disk.sdb2.in‚ 7.0546875¤BР4«*/ _wandbruntime‚12057  _runtime‚12057ú²îɬ*à trainer/global_step‚10349 % _timestamp‚1.7885224319534533e+09 _runtime‚12057.424030472 _step‚848 & lr-AdamW/pg1‚5.304830342884088e-05 & lr-AdamW/pg2‚5.304830342884088e-05ül°Î­*È & lr-AdamW/pg1‚5.304830342884088e-05 & lr-AdamW/pg2‚5.304830342884088e-05 trainer/global_step‚10349 % _timestamp‚1.7885224319534533e+09 _runtime‚12057.424030472 _step‚848Ðùs÷ô4®*/ _wandbruntime‚12057  _runtime‚12057Žú‰f(¯*€P N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 473318152 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚862966314647040 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚952420791711744 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚832199659581616 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 259638004 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12288756617216 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚862965397236224 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚632 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26090 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚62128645 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177047040 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 194984058 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1187 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 _step‚849 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 473292062 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚851198903798272 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11767410848768 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411420 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11766511440896 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 132855413 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11767410848768 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 248605696 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚843922307104207 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚851198885795328 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26088 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 259638004 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407872 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚862965397236224 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11722647522591 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1212 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 401960960 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚832199676620976 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 213654060 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955591168 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410816 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 _runtime‚12057.425057447 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955591168 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 224665600 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚843923221554987 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007241+ˆÑe (°*…P _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11767410848768 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 248605696 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚843922307104207 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚851198885795328 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26088 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 259638004 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407872 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚862965397236224 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11722647522591 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1212 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 401960960 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚832199676620976 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 213654060 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955591168 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410816 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 _runtime‚12057.425057447 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955591168 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 224665600 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚843923221554987 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007241 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 473318152 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚862966314647040 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚952420791711744 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚832199659581616 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 259638004 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12288756617216 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚862965397236224 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚632 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26090 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚62128645 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177047040 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 194984058 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1187 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 _step‚849 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 473292062 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚851198903798272 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11767410848768 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411420 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11766511440896 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 132855413ÑZ³ÀÁ4±*/ _wandbruntime‚12058  _runtime‚12058o<¾ë ²*‹N M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚844001395859653 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11723756183909 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26114 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 195002906 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 213673958 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26114 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚832277656715104 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11768523665408 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899421184 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 473362667 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚640 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 _runtime‚12058.510327933 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 259688709 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955591168 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚863047057759232 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 195002266 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 259662595 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11767624244224 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411628 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 132867791 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚851278516090880 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 473336551 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177047040 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚851278516090880 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 213673956 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 259688709 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚62134477 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 $ _timestamp‚1.788522433038954e+09 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1187 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 401960960 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 473362667 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚964799995434496 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 473336551 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚863047057759232 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚844001395859653 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11723756183909 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26114 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 195002906 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 213673958 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26114 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚832277656715104 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11768523665408 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899421184 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 473362667 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚640 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 _runtime‚12058.510327933 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 259688709 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955591168 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚863047057759232 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 195002266 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 259662595 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11767624244224 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411628 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 132867791 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚851278516090880 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 473336551 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177047040 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚851278516090880 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 213673956 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 259688709 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚62134477 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 $ _timestamp‚1.788522433038954e+09 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1187 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 401960960 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 473362667 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚964799995434496 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 473336551 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚863047057759232 PDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11822166021120 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26088 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12346747126272 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214665 _runtime‚12111.96627113 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚867002566594048 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1187 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1212 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 133474213 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 % _timestamp‚1.7885224864958544e+09 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955591168 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 trainer/global_step‚10399 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11822166021120 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177047040 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 214648960 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚867003484004864 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚969233735026176 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 235151360 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚867003484004864 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12346982277632 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 260870906 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚653 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚836098681327376 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007241 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 214648958 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 401960960 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 214648960 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 260870906 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚867002566594048 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 259091456 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 _step‚853 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚855180418575872 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 214648958 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26088 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 195892375 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚651 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚956886752748544 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11778991591119 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 B·›nt.small_pool.peak‚564 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26090 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 475545954 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410816 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚956886728808448 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚847877672918495 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚62417509 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 195891722 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 133474215 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚62418160 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚855180400572928 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171’æ (Ä*†P JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚653 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚836098681327376 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007241 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 214648958 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 401960960 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 214648960 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 260870906 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚867002566594048 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 259091456 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 _step‚853 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚855180418575872 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 214648958 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26088 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 195892375 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚651 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚956886752748544 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11778991591119 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚956886728808448 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚847877672918495 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚62417509 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 195891722 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 133474215 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚62418160 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚855180400572928 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450780 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚836098664288016 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411420 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11822166021120 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26088 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12346747126272 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214665 _runtime‚12111.96627113 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚867002566594048 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1187 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1212 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 133474213 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 % _timestamp‚1.7885224864958544e+09 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955591168 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 trainer/global_step‚10399 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11822166021120 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177047040 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 214648960 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚867003484004864 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚969233735026176 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 235151360 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚867003484004864 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12346982277632 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 260870906 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171ÕðW]4Å*/ _wandbruntime‚12113  _runtime‚121130{*'Æ*ŒN D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 401960960 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚855260048871424 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚969323982260224 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11779202841017 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 475590469 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11780100252645 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚855260030868480 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411628 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 214668856 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚867084227117056 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177047040 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 133486591 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚836176644382144 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26114 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 475564353 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 260895497 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 _runtime‚12113.078033866 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899421184 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚62423804 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 236009472 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11823278824448 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917424128 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 214668858 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 214668856 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1187 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12347934231040 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450988 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 % _timestamp‚1.7885224876055408e+09 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚867084227117056 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚969324218269696 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚956976071969280 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚956976048029184 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚855260030868480 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚62423160 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 475590469 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚646 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 195909749 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚855260048871424 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 260921611 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31686 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 trainer/global_step‚10399 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚836176644382144 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26114 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 475564353 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 260895497 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 _runtime‚12113.078033866 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899421184 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚62423804 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 236009472 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11823278824448 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917424128 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 214668858 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 214668856 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1187 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12347934231040 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450988 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 % _timestamp‚1.7885224876055408e+09 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚867084227117056 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚969324218269696 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚956976071969280 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚956976048029184 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚855260030868480 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚62423160 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 475590469 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚646 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 195909749 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚855260048871424 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 260921611 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31686 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 trainer/global_step‚10399 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 401960960 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚855260048871424 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚969323982260224 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11779202841017 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 475590469 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11780100252645 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚855260030868480 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411628 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 214668856 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0ÖB2î4È*/ _wandbruntime‚12113  _runtime‚12113ãJÈ—É*Ú 0loss/FRt2/assignment_loss‚1.4653992056846619 /loss/FRt3/assignment_loss‚2.009735334802557 0loss/FRt4/assignment_loss‚2.6038314429196445 0loss/FBt2/assignment_loss‚0.1192816113187921 0loss/FBt4/assignment_loss‚0.2200433815507075 1loss/SRqqt1/detection_loss‚1.3810585808219775 /loss/FBt2/detection_loss‚0.5951464890477884 epoch‚8 -loss/FRt1/assignment_loss‚2.508309cÁÉ56155004 /loss/FRt2/detection_loss‚1.5167852133745328 1loss/SRqqt3/detection_loss‚1.4568399459360137 .loss/FBt3/detection_loss‚2.092292050520579 &loss/total_loss‚37.811769437623084 /loss/FRt3/detection_loss‚2.6274235631442733 .loss/FRt4/detection_loss‚3.173771871761842 _step‚855 1loss/SRqqt2/detection_loss‚1.1519461972463243 2loss/SRqqt2/assignment_loss‚0.9240084102131286 2loss/SRqqt1/assignment_loss‚1.2463239133226212 /loss/FBt4/detection_loss‚2.0001453848985524 /loss/FRt1/detection_loss‚1.7642259146413652 2loss/SRqqt4/assignment_loss‚1.2794302165508271 3loss/FBt1/assignment_loss‚0.0011362514960637782 1loss/SRqqt4/detection_loss‚2.4773166527350745 /loss/FBt1/detection_loss‚3.8989145755767822 2loss/SRqqt3/assignment_loss‚1.1936941800328593 1loss/FBt3/assignment_loss‚0.10471343452571584 trainer/global_step‚10399 % _timestamp‚1.7885224876100621e+09 _runtime‚12113.080392797yÙ6ðeÊ*ß 3loss/FBt1/assignment_loss‚0.0011362514960637782 1loss/SRqqt4/detection_loss‚2.4773166527350745 /loss/FBt1/detection_loss‚3.8989145755767822 2loss/SRqqt3/assignment_loss‚1.1936941800328593 1loss/FBt3/assignment_loss‚0.10471343452571584 trainer/global_step‚10399 % _timestamp‚1.7885224876100621e+09 _runtime‚12113.080392797 0loss/FRt2/assignment_loss‚1.4653992056846619 /loss/FRt3/assignment_loss‚2.009735334802557 0loss/FRt4/assignment_loss‚2.6038314429196445 0loss/FBt2/assignment_loss‚0.1192816113187921 0loss/FBt4/assignment_loss‚0.2200433815507075 1loss/SRqqt1/detection_loss‚1.3810585808219775 /loss/FBt2/detection_loss‚0.5951464890477884 epoch‚8 -loss/FRt1/assignment_loss‚2.5083056155004 /loss/FRt2/detection_loss‚1.5167852133745328 1loss/SRqqt3/detection_loss‚1.4568399459360137 .loss/FBt3/detection_loss‚2.092292050520579 &loss/total_loss‚37.811769437623084 /loss/FRt3/detection_loss‚2.6274235631442733 .loss/FRt4/detection_loss‚3.173771871761842 _step‚855 1loss/SRqqt2/detection_loss‚1.1519461972463243 2loss/SRqqt2/assignment_loss‚0.9240084102131286 2loss/SRqqt1/assignment_loss‚1.2463239133226212 /loss/FBt4/detection_loss‚2.0001453848985524 /loss/FRt1/detection_loss‚1.7642259146413652 2loss/SRqqt4/assignment_loss‚1.2794302165508271×ø5Eû“Ë*‚(:ˆ þßêÔ·œŸ€ gpu.0.gpu‚55.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚37.0 gpu.0.powerWatts‚129.282' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚32.3205 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÿãr‹uÌ*‚(:ê þßêÔÙèÔ€ disk.rbd2.out‚ 3.529296875 disk.nvme1n1.in‚ 498.80078125" proc.memory.rssMB‚ 3001.5546875 network.sent‚ 2.7282347e+07) disk./.usagePercent‚75.87403047709512$ disk./.usageGB‚2713.622844696045 disk.rbd2.in‚0 disk.sda2.in‚ 10.7578125 disk.sdb2.in‚ 7.0546875 disk.sda.in‚ 10.7578125" disk.nvme1n1.out‚ 65728.4140625 disk.sdb1.in‚0 disk.sdb1.out‚0 cpu‚25.206933924411082! disk.dm-0.out‚134416.23046875 disk.md0p1.in‚17.8125 disk.nvme0n1.in‚ 619.63671875 disk.sdb.in‚ 7.0546875 disk.md0.in‚17.8125 disk.sda1.out‚0! disk.sdb.out‚1385.72607421875$ memory_percent‚92.11549758911133 proc.cpu.threads‚17" disk.sdb2.out‚1385.72607421875 disk.sda1.in‚0" disk.sda2.out‚1385.72607421875* proc.memory.percent‚18.320035934448242 disk.md0p1.out‚ 1335.8984375 network.recv‚ 3.906427e+06 disk.md0.out‚ 1335.8984375 disk.dm-0.in‚ 1118.4375# disk.nvme0n1.out‚68687.81640625! disk.sda.out‚1385.72607421875 disk.rbd0.in‚0 disk.rbd0.out‚ 3.37939453125' proc.memory.availableMB‚ 1291.796875$7Í*‚(:† àêÔ¢Äÿ gpu.0.gpu‚6.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚164.372' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚41.093 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ØüоoÎ*‚(:ä àêÔûÚ‚* proc.memory.percent‚18.320035934448242 disk.rbd2.out‚ 3.529296875! disk.nvme0n1.out‚ 69054.796875 disk.sdb.in‚ 7.0546875 disk.sdb1.in‚0 disk.sda1.out‚0 disk.dm-0.in‚ 1118.4375 disk.sdb.out‚1387.240234375 disk.sda.out‚1387.240234375 disk.rbd0.in‚0 disk.sdb1.out‚0 proc.cpu.threads‚17 disk.dm-0.out‚134783.3671875 disk.nvme1n1.in‚ 498.80078125" disk.nvme1n1.out‚ 65728.5703125 disk.rbd2.in‚0$ memory_percent‚92.11547374725342) proc.memory.availableMB‚ 1291.80078125 network.recv‚ 3.911317e+06 disk.md0p1.in‚17.8125 disk.nvme0n1.in‚ 619.63671875 disk.sda.in‚ 10.7578125 disk.md0.in‚17.8125 network.sent‚ 2.7378944e+07 disk.md0p1.out‚ 1337.34765625 disk.sda2.in‚ 10.7578125 disk.rbd0.out‚ 3.37939453125 disk.md0.out‚ 1337.34765625% disk./.usageGB‚2713.8738441467285 disk.sda1.in‚0 disk.sdb2.in‚ 7.0546875" proc.memory.rssMB‚ 3001.5546875) disk./.usagePercent‚75.88104852678767 disk.sda2.out‚1387.240234375 disk.sdb2.out‚1387.240234375 cpu‚25.206677175092103|<Ò¸“Ï*‚(:ˆ œàêÔÑÒÍÿ gpu.0.gpu‚38.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚37.0 gpu.0.powerWatts‚152.282' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚38.0705 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0©{0iÐ*‚(:Þ œàêÔÚ€ü€ disk.sdb.in‚ 7.0546875 disk.sda2.in‚ 10.7578125 disk.md0.out‚ 1338.859375) proc.memory.availableMB‚ 1291.80078125 network.recv‚ 3.915618e+06) disk./.usagePercent‚75.88100735582226 disk.sda.in‚ 10.7578125 disk.sdb1.out‚0 disk.dm-0.out‚134999.3515625 cpu‚25.206422793494106$ disk./.usageGB‚2713.872371673584 disk.md0p1.out‚ 1338.859375 disk.sdb.out‚ 1388.80859375 disk.rbd0.out‚ 3.37939453125 disk.sdb2.in‚ 7.0546875 disk.sda2.out‚ 1388.80859375 disk.md0p1.in‚17.8125 disk.sda1.in‚0 disk.sda1.out‚0* proc.memory.percent‚18.320035934448242 proc.cpu.threads‚17 disk.nvme1n1.in‚ 498.80078125# disk.nvme1n1.out‚65944.39453125 disk.rbd2.in‚0# disk.nvme0n1.out‚69054.95703125 disk.sda.out‚ 1388.80859375 disk.rbd0.in‚0 disk.sdb1.in‚0 disk.rbd2.out‚ 3.529296875$ memory_percent‚92.11547374725342" proc.memory.rssMB‚ 3001.5546875 network.sent‚ 2.7384923e+07 disk.nvme0n1.in‚ 619.63671875 disk.md0.in‚17.8125 disk.sdb2.out‚ 1388.80859375 disk.dm-0.in‚ 1118.4375Œgð“Ñ*‚(:ˆ «àêÔõÐ¥€ gpu.0.gpu‚47.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚37.0 gpu.0.powerWatts‚145.098' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚36.2745 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¶ð,tÒ*‚(:é «àêÔ™«û€ disk.rbd0.out‚ 3.44189453125 disk.rbd2.in‚0! disk.sdb2.out‚1391.0166015625* proc.memory.percent‚18.320035934448242 cpu‚25.206168818747905 disk.nvme0n1.out‚ 69317.46875 disk.sda.in‚ 10.7578125 disk.sda.out‚1391.0166015625# disk.nvme1n1.out‚66124.62890625 disk.rbd2.out‚ 3.529296875 disk.sdb2.in‚ 7.0546875 disk.sda1.in‚0 disk.dm-0.in‚ 1118.4375! disk.dm-0.out‚135442.09765625" proc.memory.rssMB‚ 3001.5546875 proc.cpu.threads‚17 network.sent‚ 2.7388691e+07 disk.sda2.in‚ 10.7578125 disk.md0p1.in‚17.8125 disk.nvme1n1.in‚ 498.80078125 disk.sdb1.in‚0 disk.md0.out‚ 1341.00390625) disk./.usagePercent‚75.88100799578545 disk.md0.in‚17.8125 disk.md0p1.out‚ 1341.00390625$ memory_percent‚92.11699962615967% disk./.usageGB‚2713.8723945617676 disk.sdb.out‚1391.0166015625 disk.rbd0.in‚0 disk.sda1.out‚0) proc.memory.availableMB‚ 1291.55078125 network.recv‚ 3.919815e+06! disk.sda2.out‚1391.0166015625 disk.sdb1.out‚0 disk.nvme0n1.in‚ 619.63671875 disk.sdb.in‚ 7.0546875O›Í’4Ó*/ _wandbruntime‚12169  _runtime‚12169a»š¦ÉÔ*à _runtime‚12169.695971042 _step‚856 & lr-AdamW/pg1‚4.724061304254884e-05 & lr-AdamW/pg2‚4.724061304254884e-05 trainer/global_step‚10449 % _timestamp‚1.7885225442255306e+09ùg¾ÎÕ*È _runtime‚12169.695971042 _step‚856 & lr-AdamW/pg1‚4.724061304254884e-05 & lr-AdamW/pg2‚4.724061304254884e-05 trainer/global_step‚10449 % _timestamp‚1.7885225442255306e+09تêÆ4Ö*/ _wandbruntime‚12169  _runtime‚121696)}¯(×*‚P M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 262103808 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407872 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚62705780 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1187 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚839997686033776 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11877820601344 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 262129896 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚62706367 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214665 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917410816 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 215643860 % _timestamp‚1.7885225442262516e+09 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955591168 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407872 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 215643858 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 134093015 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 215643858 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411420 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚871039735951872 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _runtime‚12169.696670875 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚973757949364736 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 _step‚857 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26088 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955591168 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚961352689845248 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚859161933353472 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚871039735951872 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚859161915350528 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 196798793 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11877820601344 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 203694080 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚859161933353472 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450780 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007241 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 215643860 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚859161915350528 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11878720009216 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917410816 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 215643860 % _timestamp‚1.7885225442262516e+09 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955591168 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407872 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 215643858 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 134093015 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 215643858 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411420 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚871039735951872 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _runtime‚12169.696670875 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚973757949364736 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 _step‚857 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26088 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955591168 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚961352689845248 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚859161933353472 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚871039735951872 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚859161915350528 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 196798793 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11877820601344 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 203694080 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚859161933353472 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450780 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007241 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 215643860 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚859161915350528 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11878720009216 JDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 196816824 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚62712040 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚62711435 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚961442009065984 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11879832825856 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚840075666127904 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 _runtime‚12170.806135739 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 134105389 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177047040 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚871120479050752 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11878933404672 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450988 _step‚858 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 1283544¸ë”3712 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899421184 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11878933404672 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 262128399 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚851910298586669 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11879832825856 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚859241563649024 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 244398080 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917424128 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 262154513 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1187 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚973848461924352 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 % _timestamp‚1.7885225453350415e+09 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 477818271 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 262154513 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 477818271 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚871121396474880 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26114 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚871121396474880 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚859241545646080 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚605 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11835546909753 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007241 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177047040 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚859241563649024 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 196817431 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 401960960 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12406208460288 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚859241545646080 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26114 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚961442033006080 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 215663758 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚973848217526272 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 215663758 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 215663756 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917424128 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 262154513 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1187 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚973848461924352 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 % _timestamp‚1.7885225453350415e+09 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 477818271 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 262154513 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 477818271 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚871121396474880 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26114 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚871121396474880 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚859241545646080 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚605 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11835546909753 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007241 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177047040 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚859241563649024 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 196817431 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 401960960 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12406208460288 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚859241545646080 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26114 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚961442033006080 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 215663758 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚973848217526272 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 215663758 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 215663756 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 196816824 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚62712040 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚62711435 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚961442009065984 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11879832825856 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚840075666127904 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 _runtime‚12170.806135739 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 134105389 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177047040 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚871120479050752 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11878933404672 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450988 _step‚858Úe¯c£4Ü*/ _wandbruntime‚12170  _runtime‚12170Õ`Ý*Ú 2loss/SRqqt1/assignment_loss‚0.9648450783434066 2loss/FBt3/assignment_loss‚0.025143971802481246 .loss/FRt4/detection_loss‚3.319347912356967 1loss/SRqqt1/detection_loss‚1.4072089904641851 .loss/FBt3/detection_loss‚3.203812849521637 %loss/total_loss‚38.76336694956801 _step‚859 % _timestamp‚1.7885225453372471e+09 0loss/FRt1/assignment_loss‚2.3422252881526946 2loss/SRqqt2/assignment_loss‚0.9006216795351661 1loss/FBt1/assignment_loss‚0.20989805704752895 0loss/FBt4/assignment_loss‚0.3019358360942811 _runtime‚12170.807531215 0loss/FRt2/assignment_loss‚1.3072906423498083 /loss/FRt4/assignment_loss‚3.482784566425142 /loss/FBt1/detection_loss‚1.8050364663726406 /loss/FBt2/detection_loss‚0.6041573358492719 epoch‚8 2loss/FBt2/assignment_loss‚0.056825956496910704 /loss/FRt3/detection_loss‚2.6052211135424472 .loss/FBt4/detection_loss‚3.168998294406467 0loss/FRt3/assignment_loss‚2.0655512100941427 1loss/SRqqt3/assignment_loss‚1.119427415241878 2loss/SRqqt4/assignment_loss‚1.8768529837640624 /loss/FRt1/detection_loss‚1.7895389125403016 1loss/SRqqt3/detection_loss‚1.3255950921485502 0loss/SRqqt4/detection_loss‚2.503191966563463 /loss/FRt2/detection_loss‚1.4849577492309942 1loss/SRqqt2/detection_loss‚0.8928975812235816 trainer/global_step‚10449žØ•œeÞ*ß /loss/FBt1/detection_loss‚1.8050364663726406 /loss/FBt2/detection_loss‚0.6041573358492719 epoch‚8 2loss/FBt2/assignment_loss‚0.056825956496910704 /loss/FRt3/detection_loss‚2.6052211135424472 .loss/FBt4/detection_loss‚3.168998294406467 0loss/FRt3/assignment_loss‚2.0655512100941427 1loss/SRqqt3/assignment_loss‚1.119427415241878 2loss/SRqqt4/assignment_loss‚1.8768529837640624 /loss/FRt1/detection_loss‚1.7895389125403016 1loss/SRqqt3/detection_loss‚1.3255950921485502 0loss/SRqqt4/detection_loss‚2.503191966563463 /loss/FRt2/detection_loss‚1.4849577492309942 1loss/SRqqt2/detection_loss‚0.8928975812235816 trainer/global_step‚10449 2loss/SRqqt1/assignment_loss‚0.9648450783434066 2loss/FBt3/assignment_loss‚0.025143971802481246 .loss/FRt4/detection_loss‚3.319347912356967 1loss/SRqqt1/detection_loss‚1.4072089904641851 .loss/FBt3/detection_loss‚3.203812849521637 %loss/total_loss‚38.76336694956801 _step‚859 % _timestamp‚1.7885225453372471e+09 0loss/FRt1/assignment_loss‚2.3422252881526946 2loss/SRqqt2/assignment_loss‚0.9006216795351661 1loss/FBt1/assignment_loss‚0.20989805704752895 0loss/FBt4/assignment_loss‚0.3019358360942811 _runtime‚12170.807531215 0loss/FRt2/assignment_loss‚1.3072906423498083 /loss/FRt4/assignment_loss‚3.482784566425142ÛÐîIî”ß*‚(:‰ ºàêÔž¹ûÿ gpu.0.gpu‚55.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚162.481' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚40.62025 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚02»!Mxà*‚(:í ºàêÔ›˜œƒ disk.rbd0.out‚ 3.44189453125# disk.nvme1n1.out‚66196.72265625! disk.sda.out‚1392.60107421875$ memory_percent‚91.59462451934814 disk.md0.in‚17.8125" disk.sdb2.out‚1392.60107421875 disk.sda1.in‚0 disk.md0p1.in‚17.8125 cpu‚25.205905578134065 disk.sdb1.out‚0 disk.sda2.in‚ 10.7578125 disk.nvme0n1.in‚ 619.63671875 disk.sdb.in‚ 7.0546875! disk.sdb.out‚1392.60107421875 proc.cpu.threads‚17 disk.rbd2.in‚0 disk.dm-0.out‚135619.0546875) proc.memory.percent‚17.77818202972412# disk.nvme0n1.out‚69422.33203125 network.sent‚ 2.7407953e+07 network.recv‚ 3.924434e+06) disk./.usagePercent‚75.88100756914334 disk.nvme1n1.in‚ 498.80078125 disk.rbd2.out‚ 3.529296875 disk.sda.in‚ 10.7578125) proc.memory.availableMB‚ 1377.13671875" disk.sda2.out‚1392.60107421875 disk.sda1.out‚0# proc.memory.rssMB‚ 2912.77734375 disk.sdb1.in‚0 disk.rbd0.in‚0 disk.md0.out‚ 1342.5234375 disk.sdb2.in‚ 7.0546875 disk.dm-0.in‚ 1118.4375 disk.md0p1.out‚ 1342.5234375% disk./.usageGB‚2713.8723793029785jŒÒ“á*‚(:ˆ ÉàêÔûäùÿ gpu.0.gpu‚44.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚169.942' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.4855 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ooâ*‚(:ä ÉàêÔõÆ”€ disk.md0p1.out‚ 1344.203125 disk.sdb.out‚1394.3427734375 disk.rbd0.in‚0 disk.rbd0.out‚ 3.4658203125 disk.md0.out‚ 1344.203125 disk.md0p1.in‚17.8125' proc.memory.availableMB‚ 1377.390625 disk.md0.in‚17.8125 disk.dm-0.in‚ 1118.4375 disk.sda.out‚1394.3427734375$ memory_percent‚91.59307479858398 cpu‚25.205645643525003# disk.nvme0n1.out‚69632.51953125 disk.sdb.in‚ 7.0546875) disk./.usagePercent‚75.86618762150768" disk.nvme1n1.out‚ 66257.9140625! disk.dm-0.out‚135890.43359375 disk.nvme0n1.in‚ 619.63671875# proc.memory.rssMB‚ 2912.77734375 network.sent‚ 2.7502042e+07 disk.sda2.in‚ 10.7578125! disk.sda2.out‚1394.3427734375 disk.sdb1.out‚0 disk.rbd2.in‚0 disk.rbd2.out‚ 3.529296875) proc.memory.percent‚17.77818202972412 proc.cpu.threads‚17! disk.sdb2.out‚1394.3427734375 disk.sda.in‚ 10.7578125 network.recv‚ 3.928868e+06 disk.nvme1n1.in‚ 498.80078125 disk.sdb2.in‚ 7.0546875% disk./.usageGB‚2713.3423461914062 disk.sdb1.in‚0 disk.sda1.in‚0 disk.sda1.out‚0y,«’ã*‚(:‡ ØàêÔÀ‹×ÿ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚37.0 gpu.0.powerWatts‚169.942' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.4855 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0òÐáBrä*‚(:ç ØàêÔûú‹€# disk.nvme1n1.out‚66258.27734375 disk.sdb1.out‚0 disk.rbd2.out‚ 3.529296875 disk.dm-0.in‚ 1118.4375 disk.md0p1.in‚17.8125 disk.nvme0n1.out‚69637.75 disk.sda2.in‚ 10.7578125 disk.rbd0.out‚ 3.4658203125 disk.nvme1n1.in‚ 498.80078125" disk.sdb2.out‚1398.24072265625$ memory_percent‚91.59533977508545 proc.cpu.threads‚17 network.sent‚ 2.7510304e+07% disk./.usageGB‚2713.3424797058105! disk.sda.out‚1398.24072265625 disk.rbd0.in‚0 disk.md0.in‚17.8125) proc.memory.availableMB‚ 1377.01953125 cpu‚25.20543466557567 disk.sdb2.in‚ 7.0546875 disk.rbd2.in‚0 disk.sda.in‚ 10.7578125" disk.sda2.out‚1398.24072265625 disk.sdb1.in‚0 disk.md0.out‚ 1348.03515625 disk.md0p1.out‚ 1348.03515625# proc.memory.rssMB‚ 2912.77734375( disk./.usagePercent‚75.8661913546263 disk.nvme0n1.in‚ 619.63671875 disk.sdb.in‚ 7.0546875! disk.sdb.out‚1398.24072265625) proc.memory.percent‚17.77818202972412 network.recv‚ 3.933532e+06 disk.sda1.in‚0 disk.sda1.out‚0! disk.dm-0.out‚135896.02734375=DŠArå*‚(:ç çàêÔ«ß°ÿ disk.rbd2.out‚ 3.529296875! disk.dm-0.out‚136109.85546875 disk.sdb.in‚ 7.0546875 disk.rbd0.in‚0 disk.sdb2.in‚ 7.0546875! disk.sdb.out‚1399.97412109375 network.recv‚ 3.937457e+06$ disk./.usageGB‚2713.279987335205 disk.md0.out‚ 1349.703125 disk.sda1.in‚0 disk.sda.in‚ 10.7578125( proc.memory.availableMB‚ 1377.0234375# proc.memory.rssMB‚ 2912.77734375 network.sent‚ 2.751622e+07 disk.md0.in‚17.8125 disk.md0p1.in‚17.8125 disk.nvme1n1.in‚ 498.80078125 cpu‚25.20518622228156 proc.cpu.threads‚17) disk./.usagePercent‚75.86444404178908 disk.rbd2.in‚0 disk.sda1.out‚0 disk.dm-0.in‚ 1118.4375 disk.md0p1.out‚ 1349.703125" disk.sda2.out‚1399.97412109375# disk.nvme1n1.out‚66259.98828125" disk.sdb2.out‚1399.97412109375 disk.nvme0n1.in‚ 619.63671875! disk.sda.out‚1399.97412109375 disk.rbd0.out‚ 3.52880859375 disk.sdb1.out‚0" disk.nvme0n1.out‚ 69849.8671875 disk.sdb1.in‚0) proc.memory.percent‚17.77818202972412 disk.sda2.in‚ 10.7578125$ memory_percent‚91.59531593322754³0X”æ*‚(:‰ çàêÔ±îÈÿ gpu.0.gpu‚16.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚36.0 gpu.0.powerWatts‚114.131' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.53275 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ä9{–4ç*/ _wandbruntime‚12228  _runtime‚12228äß{©Éè*à & lr-AdamW/pg1‚4.445575971338194e-05 & lr-AdamW/pg2‚4.445575971338194e-05 trainer/global_step‚10499 % _timestamp‚1.7885226027943685e+09 _runtime‚12228.264896581 _step‚860$×™öÎé*È _runtime‚12228.264896581 _step‚860 & lr-AdamW/pg1‚4.445575971338194e-05 & lr-AdamW/pg2‚4.445575971338194e-05 trainer/global_step‚10499 % _timestamp‚1.7885226027943685e+09Üd‹%4ê*/ _wandbruntime‚12228  _runtime‚12228Ûκ}Jë*P UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 216638760 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 197707369 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 212082688 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚843896690740176 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 216638760 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚863143448131072 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚863143430128128 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 263362798 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚855786575645511 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007241 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 216638758 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚62995556 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚875076905309696 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11933475181568 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚863143448131072 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11934374589440 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 134711815 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚978282546743808 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1212 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1187 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450780 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26088 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11889884905335 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 216638758 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 197708008 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚855785661194731 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 263336710 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 236022784 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚637 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12463659838976 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚875076905309696 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26090 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚965818674822144 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 _runtime‚12228.266370577 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚965818650882048 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11888987493915 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11934374589440 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407872 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410816 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 263362798 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚978282310721024 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 401960960 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚863143430128128 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 134711813 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 trainer/global_step‚10499 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407872 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 480001558 _step‚861 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26088 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚863143448131072 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚863143430128128 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 263362798 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚855786575645511 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007241 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 216638758 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚62995556 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚875076905309696 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11933475181568 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚863143448131072 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11934374589440 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 134711815 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚978282546743808 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1212 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1187 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450780 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26088 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11889884905335 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 216638758 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 197708008 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚855785661194731 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 263336710 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 236022784 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚637 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12463659838976 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚875076905309696 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26090 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚965818674822144 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 _runtime‚12228.266370577 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚965818650882048 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11888987493915 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11934374589440 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407872 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410816 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 263362798 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚978282310721024ÝVèèÖ4í*/ _wandbruntime‚12229  _runtime‚12229óÉãÊ'î*ŒN YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11934587984896 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31686 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚633 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 216658658 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1187 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450988 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 216658658 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚863223078426624 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177047040 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11934587984896 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚875157648408576 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26114 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11935487406080 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899421184 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 197726127 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 480046073 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177047040 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899421184 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 263361301 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚863223060423680 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12464825362432 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11890993566861 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955591168 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26114 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 216658656 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚978372795465216 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917424128 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 197725492 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411628 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 480046073 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 % _timestamp‚1.7885226039936051e+09 _step‚862 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚863223060423680 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚843974670834304 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 401960960 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚843974653794944 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 _runtime‚12229.465214922 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11935487406080 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚855864749950177 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚965907994042880 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 480019957 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 134724189 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 244398080 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 480019957 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚855865664401165 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 220457984 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚875158565832704 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 480019957 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 134724189 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 244398080 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 480019957 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚855865664401165 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 220457984 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚875158565832704 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚863223078426624 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177047040 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11934587984896 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚875157648408576 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26114 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11935487406080 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899421184 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 197726127 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 480046073 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177047040 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899421184 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 263361301 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚863223060423680 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12464825362432 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11890993566861 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955591168 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26114 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 216658656 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚978372795465216 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917424128 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 197725492 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411628 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 480046073 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 % _timestamp‚1.7885226039936051e+09 _step‚862 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚863223060423680 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚843974670834304 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 401960960 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚843974653794944 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 _runtime‚12229.465214922Þ-[yp4ð*/ _wandbruntime‚12229  _runtime‚12229MŸ‚Zñ*Ô 0loss/FRt1/assignment_loss‚2.3798693862615847 2loss/SRqqt3/assignment_loss‚1.2658332782531423 1loss/FBt2/assignment_loss‚0.03560795037551036 trainer/global_step‚10499 1loss/SRqqt4/assignment_loss‚1.255319284896056 /loss/FBt2/detection_loss‚0.6089049579160338 .loss/FBt3/detection_loss‚2.115922752022743 1loss/FBt3/assignment_loss‚0.22613213727845505 1loss/SRqqt1/detection_loss‚1.5554091320854837 /loss/SRqqt4/detection_loss‚2.93097568800052 /loss/FRt1/detection_loss‚2.1394172016432504 .loss/FRt4/detection_loss‚2.838706849018733 epoch‚8 % _timestamp‚1.7885226039961348e+09 1loss/FBt4/assignment_loss‚0.17718692674861813 %loss/total_loss‚37.74981024695717 _runtime‚12229.466477448 /loss/FRt3/assignment_loss‚2.333706097943442 /loss/FRt4/assignment_loss‚2.532560038566589 2loss/SRqqt2/assignment_loss‚0.9591460733984908 0loss/FBt1/assignment_loss‚0.2766668899870268 1loss/SRqqt2/detection_loss‚1.0641260414818923 /loss/FBt1/detection_loss‚2.4485638216137886 2loss/SRqqt1/assignment_loss‚1.2387193988302145 .loss/FRt2/detection_loss‚1.669531963345613 .loss/FRt3/detection_loss‚2.148865193128586 .loss/FBt4/detection_loss‚2.661098912358284 0loss/FRt2/assignment_loss‚1.1752493802238912 1loss/SRqqt3/detection_loss‚1.7122908915792192 _step‚863ݨ«a_ò*Ù 1loss/FBt4/assignment_loss‚0.17718692674861813 %loss/total_loss‚37.74981024695717 _runtime‚12229.466477448 /loss/FRt3/assignment_loss‚2.333706097943442 /loss/FRt4/assignment_loss‚2.532560038566589 2loss/SRqqt2/assignment_loss‚0.9591460733984908 0loss/FBt1/assignment_loss‚0.2766668899870268 1loss/SRqqt2/detection_loss‚1.0641260414818923 /loss/FBt1/detection_loss‚2.4485638216137886 2loss/SRqqt1/assignment_loss‚1.2387193988302145 .loss/FRt2/detection_loss‚1.669531963345613 .loss/FRt3/detection_loss‚2.148865193128586 .loss/FBt4/detection_loss‚2.661098912358284 0loss/FRt2/assignment_loss‚1.1752493802238912 1loss/SRqqt3/detection_loss‚1.7122908915792192 _step‚863 0loss/FRt1/assignment_loss‚2.3798693862615847 2loss/SRqqt3/assignment_loss‚1.2658332782531423 1loss/FBt2/assignment_loss‚0.03560795037551036 trainer/global_step‚10499 1loss/SRqqt4/assignment_loss‚1.255319284896056 /loss/FBt2/detection_loss‚0.6089049579160338 .loss/FBt3/detection_loss‚2.115922752022743 1loss/FBt3/assignment_loss‚0.22613213727845505 1loss/SRqqt1/detection_loss‚1.5554091320854837 /loss/SRqqt4/detection_loss‚2.93097568800052 /loss/FRt1/detection_loss‚2.1394172016432504 .loss/FRt4/detection_loss‚2.838706849018733 epoch‚8 % _timestamp‚1.7885226039961348e+09ߥ8àžó*‚(:“ öàêÔì½éý gpu.0.gpu‚48.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚36.0 gpu.0.powerWatts‚119.368' gpu.0.enforcedPowerLimitWatts‚400.0) gpu.0.powerPercent‚29.841999999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0퀚½xô*‚(:í öàêԫĘ€! disk.sda.out‚1401.79345703125 disk.sda2.in‚ 10.7578125 disk.nvme1n1.in‚ 498.80078125 disk.rbd2.in‚0 disk.rbd2.out‚ 3.529296875 disk.md0p1.out‚ 1351.45703125$ memory_percent‚91.59722328186035" disk.sda2.out‚1401.79345703125 disk.rbd0.in‚0 disk.sdb1.in‚0( proc.memory.availableMB‚ 1376.7109375# proc.memory.rssMB‚ 2912.77734375) proc.memory.percent‚17.77818202972412 disk.sdb2.in‚ 7.0546875 disk.md0p1.in‚17.8125! disk.sdb.out‚1401.79345703125 cpu‚25.204952180623717 network.sent‚ 2.7520337e+07 disk.md0.in‚17.8125 disk.md0.out‚ 1351.45703125 disk.sda1.in‚0 proc.cpu.threads‚17) disk./.usagePercent‚75.86695760388906 disk.rbd0.out‚ 3.52880859375# disk.nvme0n1.out‚69941.76171875 disk.sdb.in‚ 7.0546875 disk.sdb1.out‚0 disk.dm-0.out‚136201.9140625 disk.nvme0n1.in‚ 619.63671875 network.recv‚ 3.942216e+06# disk.nvme1n1.out‚66260.15234375 disk.sda1.out‚0 disk.dm-0.in‚ 1118.4375 disk.sda.in‚ 10.7578125" disk.sdb2.out‚1401.79345703125$ disk./.usageGB‚2713.369884490967snåù“õ*‚(:ˆ …áêÔƒÃÿ gpu.0.gpu‚53.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚37.0 gpu.0.powerWatts‚134.298' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚33.5745 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚03Š`ö*‚(:Õ …áêÔ¿îöÿ network.recv‚ 3.9471e+06 disk.rbd0.in‚0 disk.nvme1n1.in‚ 498.80078125 disk.sdb1.in‚0 disk.rbd2.out‚ 3.529296875 disk.sda1.out‚0 disk.md0p1.out‚ 1352.9765625# disk.nvme0n1.out‚70347.80859375! disk.dm-0.out‚136608.33984375 disk.sdb.out‚ 1403.3828125# proc.memory.rssMB‚ 2912.77734375) disk./.usagePercent‚75.86878544542803 disk.sda2.in‚ 10.7578125 disk.sda2.out‚ 1403.3828125 disk.sdb1.out‚0 disk.sdb.in‚ 7.0546875 disk.sda.out‚ 1403.3828125 disk.md0.out‚ 1352.9765625 disk.dm-0.in‚ 1118.4375 disk.nvme0n1.in‚ 619.63671875 proc.cpu.threads‚17 disk.nvme1n1.out‚ 66260.53125 disk.md0.in‚17.8125 network.sent‚ 2.7627208e+07$ disk./.usageGB‚2713.435256958008 disk.sda.in‚ 10.7578125 disk.rbd0.out‚ 3.591796875 disk.sdb2.out‚ 1403.3828125 disk.sda1.in‚0 disk.md0p1.in‚17.8125) proc.memory.percent‚17.77818202972412 cpu‚25.204764012378327 disk.rbd2.in‚0 disk.sdb2.in‚ 7.0546875$ memory_percent‚91.59722328186035( proc.memory.availableMB‚ 1376.7109375÷óS«”÷*‚(:‰ ”áêÔ»¸—ÿ gpu.0.gpu‚44.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚159.419' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚39.85475 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0±h2\rø*‚(:ç ”áêÔÙ³Ïÿ disk.rbd0.out‚ 3.591796875 disk.sdb2.in‚ 7.0546875 disk.sda1.out‚0 disk.sda2.in‚ 10.7578125( proc.memory.availableMB‚ 1376.7109375 cpu‚25.204535919827965 disk.sdb.in‚ 7.0546875! disk.sdb.out‚1404.81298828125) disk./.usagePercent‚75.87404935600931 disk.nvme1n1.in‚ 498.80078125 disk.md0.out‚ 1354.33984375 disk.md0p1.in‚17.8125$ memory_percent‚91.59722328186035 disk.rbd0.in‚0# disk.nvme1n1.out‚66260.72265625" disk.sdb2.out‚1404.81298828125 disk.dm-0.in‚ 1118.4375! disk.dm-0.out‚136608.53515625" disk.sda2.out‚1404.81298828125) proc.memory.percent‚17.77818202972412 disk.md0p1.out‚ 1354.33984375! disk.sda.out‚1404.81298828125 proc.cpu.threads‚17 disk.rbd2.in‚0 disk.sda.in‚ 10.7578125 network.sent‚ 2.7635572e+07 disk.sdb1.in‚0 disk.nvme0n1.in‚ 619.63671875 disk.nvme0n1.out‚ 70347.8125 disk.sdb1.out‚0 disk.rbd2.out‚ 3.529296875 disk.sda1.in‚0 network.recv‚ 3.95203e+06$ disk./.usageGB‚2713.623519897461 disk.md0.in‚17.8125# proc.memory.rssMB‚ 2912.77734375ØÛ½‘ù*‚(:† £áêÔËÞ“€ gpu.0.gpu‚3.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚166.656' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚41.664 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0+…wú*‚(:ì £áêÔÚ×€) proc.memory.availableMB‚ 1376.95703125) proc.memory.percent‚17.77818202972412 disk.dm-0.in‚ 1118.4375! disk.sdb.out‚1406.77392578125 network.sent‚ 2.7639173e+07 disk.sdb.in‚ 7.0546875 disk.rbd2.in‚0 disk.sdb2.in‚ 7.0546875 cpu‚25.204322779806393 disk.sda1.in‚0 disk.sdb1.out‚0 disk.md0.in‚17.8125 disk.rbd2.out‚ 3.529296875 disk.sda2.in‚ 10.7578125 disk.rbd0.out‚ 3.591796875 disk.nvme1n1.in‚ 498.80078125" disk.sdb2.out‚1406.77392578125# proc.memory.rssMB‚ 2912.77734375 network.recv‚ 3.956059e+06% disk./.usageGB‚2713.8737754821777 disk.sda1.out‚0 disk.md0p1.in‚17.8125 disk.md0p1.out‚ 1356.24609375 disk.sdb1.in‚0 proc.cpu.threads‚17" disk.sda2.out‚1406.77392578125) disk./.usagePercent‚75.88104660689808 disk.sda.in‚ 10.7578125! disk.sda.out‚1406.77392578125 disk.rbd0.in‚0# disk.nvme1n1.out‚66260.93359375 disk.dm-0.out‚ 137034.296875 disk.nvme0n1.in‚ 619.63671875# disk.nvme0n1.out‚70773.36328125 disk.md0.out‚ 1356.24609375$ memory_percent‚91.59572124481201±__£4û*/ _wandbruntime‚12286  _runtime‚12286¥¾YÉü*à _step‚864 & lr-AdamW/pg1‚4.175101363030886e-05 & lr-AdamW/pg2‚4.175101363030886e-05 trainer/global_step‚10549 % _timestamp‚1.7885226613770447e+09 _runtime‚12286.847382777i¢®·Îý*È _runtime‚12286.847382777 _step‚864 & lr-AdamW/pg1‚4.175101363030886e-05 & lr-AdamW/pg2‚4.175101363030886e-05 trainer/global_step‚10549 % _timestamp‚1.7885226613770447e+09àìnx÷4þ*/ _wandbruntime‚12286  _runtime‚12286!Lå(ÿ*‚P G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11944434151023 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26090 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 264595700 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚859741027009019 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚982806910984704 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26088 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12522275125760 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12522056751616 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 242314240 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚620 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11989129761792 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 218374144 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚847795695446576 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917410816 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11990029169664 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚970284611918848 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407872 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11989129761792 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 217633660 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚970284635858944 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 198617288 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚879114992078336 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11990029169664 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚63287295 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚847795678407216 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411420 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 135330615 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26088 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚867124944905728 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚867124962908672 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11945331562443 _runtime‚12286.848412782 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410816 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 264595700 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 135330613 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚63286675 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚879114074667520 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955591168 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 217633658 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _step‚865 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚622 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 217633660 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214665 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007241 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 198617288 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚879114992078336 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11990029169664 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚63287295 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚847795678407216 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411420 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 135330615 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26088 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚867124944905728 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚867124962908672 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11945331562443 _runtime‚12286.848412782 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410816 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 264595700 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 135330613 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚63286675 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚879114074667520 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955591168 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 217633658 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _step‚865 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚622 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 217633660 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214665 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007241 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11944434151023 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26090 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 264595700 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚859741027009019 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚982806910984704 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26088 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12522275125760 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12522056751616ᬛÒ4+/ _wandbruntime‚12288  _runtime‚12288 ¨ tß‚+‹N KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 198636078 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚982897149095936 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚879194817766400 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 217653558 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12523442608640 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚970373955079680 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11946440223969 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917424128 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 217653558 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 264594203 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚632 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1187 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 264620317 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11991141986304 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11991141986304 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 401960960 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177047040 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 135342991 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 217653556 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚847873658501344 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚867204593204224 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12523217956352 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450988 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚982897397688320 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 $ _timestamp‚1.788522662545217e+09 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11990242565120 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899421184 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 482247759 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 trainer/global_step‚10549 _runtime‚12288.016185963 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 482273875 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 482273875 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 264594203 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚859819201313685 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411628 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚879195735190528 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31686 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 198635444 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚847873675540704 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007241./yÝ'ƒ+N SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 135342989 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11945542812341 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 482247759 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚879194817766400 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 217653558 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12523442608640 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚970373955079680 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11946440223969 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917424128 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 217653558 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 264594203 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚632 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1187 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 264620317 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11991141986304 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11991141986304 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 401960960 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177047040 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 135342991 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 217653556 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚847873658501344 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚867204593204224 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12523217956352 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450988 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚982897397688320 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 $ _timestamp‚1.788522662545217e+09 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11990242565120 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899421184 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 482247759 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 trainer/global_step‚10549 _runtime‚12288.016185963 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 482273875 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 482273875 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 264594203 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚859819201313685 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411628 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚879195735190528 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31686 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 198635444 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚847873675540704 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007241 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 198636078 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚982897149095936 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚883152161436160 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 484457162 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚883151244025344 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚871106477686272 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 218628558 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 135949413 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚63575314 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917410816 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12045683749888 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚883152161436160 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1212 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 _runtime‚12345.149985118 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955591168 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 135949415 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450780 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955591168 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 401960960 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚63575967 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚871106459683328 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 VGDeviceStatsMonitor.on_train_batch_start›X¶º½/requested_bytes.small_pool.peak‚ 1175007241 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12044784342016 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚974750572955648 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26090 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 218628558 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12044784342016 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 $ _timestamp‚1.788522719679554e+09 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12045683749888 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26088 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚851694683113616 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407872 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12580482104832 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214665 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚863695478372527 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607H¤Q (”+†P GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26088 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚851694683113616 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407872 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12580482104832 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214665 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚863695478372527 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 484431072 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12000778219551 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚653 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 265802514 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚974750596895744 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 265828602 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26088 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚883152161436160 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 484457162 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚883151244025344 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚871106477686272 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 218628558 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 135949413 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚63575314 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917410816 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12045683749888 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚883152161436160 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1212 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 _runtime‚12345.149985118 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955591168 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 135949415 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450780 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955591168 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 401960960 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚63575967 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚871106459683328 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007241 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12044784342016 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚974750572955648 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26090 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 218628558 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12044784342016 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 $ _timestamp‚1.788522719679554e+09 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12045683749888 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712åÁ˜ç¾4•+/ _wandbruntime‚12346  _runtime‚12346 ?íI'–+‹N EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1187 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 484475561 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚883232904548352 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚871186107981824 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚638 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899421184 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 218648458 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚851772680247104 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12046796566528 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 199543448 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917424128 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 218648456 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚63581657 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411628 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 484501677 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 $ _timestamp‚1.788522720681199e+09 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 265827105 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 265853219 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚63581019 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26114 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 265827105 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12581408043520 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12000989469449 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 484475561 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚640 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚987421548812288 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚871186089978880 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚863773652677193 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚851772663207744 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12001886881077 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 199542808 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 135961789 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 224652288 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12581632695808 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 265853219 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚871186107981824 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177047040 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899421184 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚12346.152253752 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31686 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚974839916116480 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 248592384 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 401960960 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607^Ï 'Ï—+N TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007241 _step‚870 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955591168 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1187 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 484475561 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚883232904548352 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚871186107981824 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚638 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899421184 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 218648458 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚851772680247104 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12046796566528 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 199543448 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917424128 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 218648456 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚63581657 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411628 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 484501677 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 $ _timestamp‚1.788522720681199e+09 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 265827105 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 265853219 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚63581019 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26114 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 265827105 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12581408043520 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12000989469449 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 484475561 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚640 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚987421548812288 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚871186089978880 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚863773652677193 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚851772663207744 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12001886881077 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 199542808 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 135961789 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 224652288 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12581632695808 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 265853219 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚871186107981824 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177047040 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899421184 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚12346.152253752 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31686 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚974839916116480 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 248592384 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 401960960 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 EDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1212 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 200432818 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407872 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 267061504 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12638896785920 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 trainer/global_step‚10649 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 486658874 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 248605696 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450780 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12100438922240 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 401960960 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚979216533992448 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚887188413383168 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 486684964 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚63864603 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚991855206112768 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12056224876659 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955591168 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚887189330793984 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 219623460 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚875087974460928 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚875087974460928 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 224665600 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 401960960 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚979216533992448 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚887188413383168 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 486684964 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚63864603 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚991855206112768 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12056224876659 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955591168 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚887189330793984 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 219623460 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚875087974460928 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚875087974460928 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 224665600 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1212éÕÃŽ4§+/ _wandbruntime‚12404  _runtime‚12404üqœê¨+ŒN L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 486729479 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚991945939479552 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12057333538185 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26114 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.smallzÚæ(%_pool.allocated‚12102451146752 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12102451146752 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 244398080 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚875167604756480 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 267086121 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411628 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 136580589 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12101551725568 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 219643356 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 219643358 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚875167622759424 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚875167622759424 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚979305877153280 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450988 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 267086121 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12640062326272 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 267060007 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 267060007 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 200450870 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177047040 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 486703363 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 401960960 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚979305853213184 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚867728104040701 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚875167604756480 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 219643358 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7885227788028703e+09 _runtime‚12404.273818655 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚645 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 220457984 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 200450223 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12639841868288 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177047040 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚855671667914144 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 _step‚874 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 486729479 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26114 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚63869634 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 219643356 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1187 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955591168 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 trainer/global_step‚10649 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚887270073906176 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12101551725568 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚647 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 486703363 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607g 4'©+‘N XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 244398080 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚875167604756480 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 267086121 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411628 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 136580589 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12101551725568 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 219643356 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 219643358 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚875167622759424 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚875167622759424 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚979305877153280 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450988 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 267086121 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12640062326272 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 267060007 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 267060007 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 200450870 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177047040 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 486703363 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 401960960 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚979305853213184 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚867728104040701 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚875167604756480 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 219643358 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7885227788028703e+09 _runtime‚12404.273818655 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚645 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 220457984 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 200450223 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12639841868288 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177047040 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚855671667914144 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 _step‚874 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 486729479 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26114 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚63869634 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 219643356 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1187 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955591168 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 trainer/global_step‚10649 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚887270073906176 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12101551725568 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚647 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 486703363 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 486729479 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚991945939479552 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12057333538185 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26114 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12102451146752 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12102451146752êUqc=4ª+/ _wandbruntime‚12404  _runtime‚12404ë³h^«+Ø 2loss/SRqqt2/assignment_loss‚0.8773788589304925 1loss/SRqqt3/assignment_loss‚1.241017491243441 1loss/FBt2/assignment_loss‚0.15243334742539447 0loss/FBt3/assignment_loss‚0.1786269976289207 /loss/FRt3/detection_loss‚2.1780797797862603 /loss/FRt4/detection_loss‚3.5426901233823678 /loss/FBt2/detection_loss‚0.5228454233033628 epoch‚8 0loss/FRt2/assignment_loss‚1.5431612214216819 1loss/SRqqt2/detection_loss‚1.0481896430127702 /loss/FBt4/detection_loss‚2.3523363926831413 0loss/FBt4/assignment_loss‚0.2258868647670223 1loss/SRqqt4/detection_loss‚2.3027578247711062 %loss/total_loss‚37.81229956804278 _runtime‚12404.274769129 0loss/FRt3/assignment_loss‚1.8013484374336575 0loss/SRqqt1/detection_loss‚1.665233799712915 0loss/FRt1/assignment_loss‚2.5652665433130766 2loss/SRqqt1/assignment_loss‚1.1349252649747275 /loss/FRt2/detection_loss‚1.5175738901329727 trainer/global_step‚10649 /loss/FBt1/detection_loss‚2.4661799669265747 # _timestamp‚1.78852277880461e+09 _step‚875 1loss/FBt1/assignment_loss‚0.30053196577500785 /loss/FRt1/detection_loss‚1.9462715165691886 0loss/SRqqt3/detection_loss‚1.433169961720705 /loss/FBt3/detection_loss‚2.8334783485957553 0loss/FRt4/assignment_loss‚2.5750682134377327 2loss/SRqqt4/assignment_loss‚1.4078476910945028”#—c¬+Ý 1loss/FBt1/assignment_loss‚0.30053196577500785 /loss/FRt1/detection_loss‚1.9462715165691886 0loss/SRqqt3/detection_loss‚1.433169961720705 /loss/FBt3/detection_loss‚2.8334783485957553 0loss/FRt4/assignment_loss‚2.5750682134377327 2loss/SRqqt4/assignment_loss‚1.4078476910945028 2loss/SRqqt2/assignment_loss‚0.8773788589304925 1loss/SRqqt3/assignment_loss‚1.241017491243441 1loss/FBt2/assignment_loss‚0.15243334742539447 0loss/FBt3/assignment_loss‚0.1786269976289207 /loss/FRt3/detection_loss‚2.1780797797862603 /loss/FRt4/detection_loss‚3.5426901233823678 /loss/FBt2/detection_loss‚0.5228454233033628 epoch‚8 0loss/FRt2/assignment_loss‚1.5431612214216819 1loss/SRqqt2/detection_loss‚1.0481896430127702 /loss/FBt4/detection_loss‚2.3523363926831413 0loss/FBt4/assignment_loss‚0.2258868647670223 1loss/SRqqt4/detection_loss‚2.3027578247711062 %loss/total_loss‚37.81229956804278 _runtime‚12404.274769129 0loss/FRt3/assignment_loss‚1.8013484374336575 0loss/SRqqt1/detection_loss‚1.665233799712915 0loss/FRt1/assignment_loss‚2.5652665433130766 2loss/SRqqt1/assignment_loss‚1.1349252649747275 /loss/FRt2/detection_loss‚1.5175738901329727 trainer/global_step‚10649 /loss/FBt1/detection_loss‚2.4661799669265747 # _timestamp‚1.78852277880461e+09 _step‚875ë¤Qƒ”­+‚(:‰ ›âêÔ±ŸËÿ gpu.0.gpu‚36.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚163.257' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚40.81425 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‡,þñl®+‚(:á ›âêÔ ¦æ disk.dm-0.out‚ 138843.09375 network.recv‚ 3.991948e+06) disk./.usagePercent‚75.86704623879126 disk.sdb.in‚ 7.0546875 disk.sda.in‚ 10.7578125 disk.nvme1n1.in‚ 498.80078125 disk.sdb2.in‚ 7.0546875 disk.sda1.in‚0 disk.sdb.out‚1419.9951171875 disk.sda2.in‚ 10.7578125! disk.sda2.out‚1419.9951171875 disk.sdb1.out‚0 disk.md0.in‚17.8125! disk.sdb2.out‚1419.9951171875 disk.nvme0n1.in‚ 619.63671875! disk.nvme0n1.out‚ 72580.390625 disk.sda.out‚1419.9951171875! disk.nvme1n1.out‚ 66262.703125 disk.rbd2.out‚ 3.529296875 disk.dm-0.in‚ 1118.4375) proc.memory.percent‚17.77818202972412 cpu‚25.202647154249775 proc.cpu.threads‚17 network.sent‚ 2.7901784e+07 disk.sdb1.in‚0 disk.md0p1.in‚17.8125) proc.memory.availableMB‚ 1375.98828125# proc.memory.rssMB‚ 2912.77734375 disk.rbd0.out‚ 3.591796875$ memory_percent‚91.60163402557373 disk.rbd0.in‚0 disk.md0.out‚ 1369.0234375 disk.rbd2.in‚0 disk.sda1.out‚0 disk.md0p1.out‚ 1369.0234375% disk./.usageGB‚2713.3730545043945!¶Ã„”¯+‚(:‰ ªâêÔÌËÂÿ gpu.0.gpu‚30.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚37.0 gpu.0.powerWatts‚162.813' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚40.70325 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0UËè‚n°+‚(:ã ªâêÔë” € disk.sdb.in‚ 7.0546875 disk.sda2.in‚ 10.7578125& proc.memory.availableMB‚ 1376.53125 proc.cpu.threads‚17 disk.nvme1n1.in‚ 498.80078125 disk.md0p1.in‚17.8125 network.recv‚ 3.997954e+06 disk.dm-0.out‚138882.2421875! disk.sdb.out‚1421.54541015625! disk.sda.out‚1421.54541015625 disk.md0.out‚ 1370.50390625 cpu‚25.20244966935214 network.sent‚ 2.7996892e+07) disk./.usagePercent‚75.87054566418963% disk./.usageGB‚2713.4982109069824 disk.sda.in‚ 10.7578125 disk.rbd0.in‚0" disk.nvme1n1.out‚ 66262.8671875 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.sda1.in‚0 disk.md0p1.out‚ 1370.50390625$ memory_percent‚91.59832000732422# proc.memory.rssMB‚ 2912.77734375) proc.memory.percent‚17.77818202972412 disk.sdb2.in‚ 7.0546875" disk.sdb2.out‚1421.54541015625 disk.dm-0.in‚ 1118.4375 disk.rbd2.out‚ 3.529296875 disk.sda1.out‚0 disk.md0.in‚17.8125 disk.rbd2.in‚0 disk.nvme0n1.in‚ 619.63671875 disk.nvme0n1.out‚ 72619.375" disk.sda2.out‚1421.54541015625 disk.rbd0.out‚ 3.5917968757›H瑱+‚(:† ¹âêԔݙ€ gpu.0.gpu‚57.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚36.0 gpu.0.powerWatts‚114.74' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.685 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ýi>Ow²+‚(:ì ¹âêÔ›‹ç disk.md0.out‚ 1372.10546875 disk.rbd2.in‚0! disk.sdb.out‚1423.21240234375 disk.sda.in‚ 10.7578125 disk.nvme1n1.in‚ 498.80078125) proc.memory.availableMB‚ 1376.27734375) proc.memory.percent‚17.77818202972412 cpu‚25.20223948762849 disk.md0.in‚17.8125 disk.sda1.in‚0 disk.sdb.in‚ 7.0546875! disk.sda.out‚1423.21240234375 disk.sda2.in‚ 10.7578125 disk.sdb2.in‚ 7.0546875 disk.md0p1.out‚ 1372.10546875 disk.rbd0.out‚ 3.591796875 disk.sdb1.in‚0 disk.rbd2.out‚ 3.529296875" disk.sdb2.out‚1423.21240234375! disk.dm-0.out‚138946.83203125 disk.md0p1.in‚17.8125 disk.nvme0n1.in‚ 619.63671875# proc.memory.rssMB‚ 2912.77734375 network.recv‚ 4.001259e+06 disk.sdb1.out‚0$ memory_percent‚91.59986972808838# disk.nvme0n1.out‚72683.71484375 disk.rbd0.in‚0" disk.nvme1n1.out‚ 66263.1171875) disk./.usagePercent‚75.87405522233858% disk./.usageGB‚2713.6237297058105 disk.dm-0.in‚ 1118.4375" disk.sda2.out‚1423.21240234375 proc.cpu.threads‚17 network.sent‚ 2.8004357e+07 disk.sda1.out‚0]¨l³+‚(:á ÈâêÔ®óŒÿ network.sent‚ 2.8008299e+07) disk./.usagePercent‚75.88104756684288 disk.sda2.out‚1425.001953125 disk.dm-0.in‚ 1118.4375) proc.memory.availableMB‚ 1376.02734375 cpu‚25.202028113066394 disk.md0p1.out‚ 1373.83203125 disk.sda.in‚ 10.7578125 disk.sda2.in‚ 10.7578125 disk.rbd0.in‚0 disk.sdb2.out‚1425.001953125$ memory_percent‚91.60139560699463) proc.memory.percent‚17.77820587158203 disk.md0p1.in‚17.8125# disk.nvme0n1.out‚73113.95703125 disk.sdb.out‚1425.001953125 disk.md0.out‚ 1373.83203125 disk.sdb2.in‚ 7.0546875! disk.dm-0.out‚139377.26171875 disk.sda.out‚1425.001953125 disk.sdb1.out‚0 disk.rbd2.in‚0 disk.sda1.out‚0$ disk./.usageGB‚2713.873809814453 disk.sdb.in‚ 7.0546875 disk.md0.in‚17.8125 disk.nvme1n1.in‚ 498.80078125" disk.nvme1n1.out‚ 66263.3046875 proc.memory.rssMB‚ 2912.78125 network.recv‚ 4.005811e+06 disk.rbd0.out‚ 3.591796875 proc.cpu.threads‚17 disk.nvme0n1.in‚ 619.63671875 disk.sdb1.in‚0 disk.rbd2.out‚ 3.529296875 disk.sda1.in‚0Í•Ïu”´+‚(:‰ ÈâêÔ‡øïÿ gpu.0.gpu‚51.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚171.501' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.87525 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0å­ 4µ+/ _wandbruntime‚12461  _runtime‚12461^öÎɶ+à & lr-AdamW/pg1‚3.412297133550104e-05 & lr-AdamW/pg2‚3.412297133550104e-05 trainer/global_step‚10699 % _timestamp‚1.7885228363328164e+09 _runtime‚12461.803136528 _step‚876uºëêη+È & lr-AdamW/pg1‚3.412297133550104e-05 & lr-AdamW/pg2‚3.412297133550104e-05 trainer/global_step‚10699 % _timestamp‚1.7885228363328164e+09 _runtime‚12461.803136528 _step‚876ìey¾4¸+/ _wandbruntime‚12461  _runtime‚12461]ªPE¹+‚P MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚891226500151808 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 268268318 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 220618358 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚879069489238528 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚871603466648763 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚879069507241472 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12111671533767 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411420 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007241 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 228859904 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚891225582740992 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 488886676 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12110774122347 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚64154873 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 268268318 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955591168 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1212 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12697019258368 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450780 _step‚877 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚636 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚879069507241472 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚879069489238528 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12156992910336 % _timestamp‚1.7885228363338609e+09 QCDeviceStatsMonitorÓŸ¦éÃ.on_train_batch_start/allocation.large_pool.freed‚ 220618358 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12156093502464 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 201342524 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917410816 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 137187015 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 201341886 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 137187013 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 252800000 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407872 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 268294406 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 401960960 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚983682495029248 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 488912766 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26088 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚996379767087616 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚871604381099543 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚859492692526416 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177047040 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955591168 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410816 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1187 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 _runtime‚12461.804270493 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12156992910336 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 268294406 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚891226500151808 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚891225582740992 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 220618360 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚983682518969344 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407872 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚638 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚64155509 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214665 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26090 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚859492709565776 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0>‡¾‚ (º+‡P \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚879069489238528 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚871603466648763 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚879069507241472 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12111671533767 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411420 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007241 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 228859904 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚891225582740992 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 488886676 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12110774122347 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚64154873 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 268268318 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955591168 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1212 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12697019258368 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450780 _step‚877 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚636 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚879069507241472 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚879069489238528 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12156992910336 % _timestamp‚1.7885228363338609e+09 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 220618358 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12156093502464 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 201342524 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917410816 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 137187015 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 201341886 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 137187013 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 252800000 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407872 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 268294406 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 401960960 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚983682495029248 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 488912766 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26088 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚996379767087616 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚871604381099543 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚859492692526416 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177047040 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955591168 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410816 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1187 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 _runtime‚12461.804270493 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12156992910336 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 268294406 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚891226500151808 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚891225582740992 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 220618360 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚983682518969344 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407872 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚638 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚64155509 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214665 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26090 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚859492709565776 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚891226500151808 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 268268318 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 220618358íÈÞ\Ù4»+/ _wandbruntime‚12462  _runtime‚12462VD>'¼+ŠN VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚64161344 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 268319023 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 201360735 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 488931165 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899421184 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚891307243264000 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚996470011076608 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12111882783665 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 137199389 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚871683469855197 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚879149119534080 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1187 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚859570689659904 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 220638256 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚891307243264000 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26114 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚983771814249984 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450988 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 220638256 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177047040 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955591168 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12157206305792 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177047040 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 488957281 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 240203776 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 488931165 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007241 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12158105726976 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 137199391 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 220638258 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 268319023 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899421184 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚879149119534080 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚64160703 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411628 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26114 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 _step‚878 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31686 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12112780195293 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12158105726976 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 201360092 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 216263680 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 488957281 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚996470251280384 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚859570672620544 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 268292909 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚643ù&v ɽ+N UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 137199389 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚871683469855197 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚879149119534080 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1187 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚859570689659904 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 220638256 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚891307243264000 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26114 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚983771814249984 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450988 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 220638256 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177047040 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955591168 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12157206305792 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177047040 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 488957281 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 240203776 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 488931165 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007241 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12158105726976 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 137199391 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 220638258 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 268319023 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899421184 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚879149119534080 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚64160703 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411628 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26114 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 _step‚878 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31686 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12112780195293 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12158105726976 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 201360092 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 216263680 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 488957281 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚996470251280384 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚859570672620544 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 268292909 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚643 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚64161344 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 268319023 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 201360735 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 488931165 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899421184 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚891307243264000 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚996470011076608 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12111882783665 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712î•ï{4¾+/ _wandbruntime‚12462  _runtime‚12462qõç^¿+Ø 0loss/FRt2/assignment_loss‚0.8551921987015269 2loss/SRqqt4/assignment_loss‚1.4378388995038611 1loss/FBt4/assignment_loss‚0.15859060243019485 _runtime‚12462.984505169 trainer/global_step‚10699 /loss/FRt1/detection_loss‚2.3297939362940574 ,loss/FRt3/detection_loss‚2.2471730997381 /loss/FBt2/detection_loss‚0.4157436769219893 epoch‚8 /loss/FRt3/assignment_loss‚2.358759379905203 2loss/SRqqt3/assignment_loss‚0.9917044936880177 0loss/SRqqt2/detection_loss‚0.878972887141364 -loss/FBt3/detection_loss‚2.36558056473732 /loss/FRt4/detection_loss‚2.8399127292633057 1loss/SRqqt1/detection_loss‚1.6321428421139716 1loss/SRqqt4/detection_loss‚2.4903726609689847 2loss/SRqqt2/assignment_loss‚0.8259107482353492 1loss/FBt2/assignment_loss‚0.13199835769178825 2loss/FBt3/assignment_loss‚0.052113621363878336 /loss/FRt2/detection_loss‚1.7036469058012185 /loss/FBt1/detection_loss‚2.0287289809096944 % _timestamp‚1.7885228375143435e+09 0loss/FRt4/assignment_loss‚2.6990785789489746 2loss/SRqqt1/assignment_loss‚1.2395156711898745 1loss/FBt1/assignment_loss‚0.13417209616016573 1loss/SRqqt3/detection_loss‚1.6480174449582894 /loss/FBt4/detection_loss‚2.3237184584140778 %loss/total_loss‚36.32003984589379 _step‚879 /loss/FRt1/assignment_loss‚2.531361010812578p«7àcÀ+Ý 0loss/FRt4/assignment_loss‚2.6990785789489746 2loss/SRqqt1/assignment_loss‚1.2395156711898745 1loss/FBt1/assignment_loss‚0.13417209616016573 1loss/SRqqt3/detection_loss‚1.6480174449582894 /loss/FBt4/detection_loss‚2.3237184584140778 %loss/total_loss‚36.32003984589379 _step‚879 /loss/FRt1/assignment_loss‚2.531361010812578 0loss/FRt2/assignment_loss‚0.8551921987015269 2loss/SRqqt4/assignment_loss‚1.4378388995038611 1loss/FBt4/assignment_loss‚0.15859060243019485 _runtime‚12462.984505169 trainer/global_step‚10699 /loss/FRt1/detection_loss‚2.3297939362940574 ,loss/FRt3/detection_loss‚2.2471730997381 /loss/FBt2/detection_loss‚0.4157436769219893 epoch‚8 /loss/FRt3/assignment_loss‚2.358759379905203 2loss/SRqqt3/assignment_loss‚0.9917044936880177 0loss/SRqqt2/detection_loss‚0.878972887141364 -loss/FBt3/detection_loss‚2.36558056473732 /loss/FRt4/detection_loss‚2.8399127292633057 1loss/SRqqt1/detection_loss‚1.6321428421139716 1loss/SRqqt4/detection_loss‚2.4903726609689847 2loss/SRqqt2/assignment_loss‚0.8259107482353492 1loss/FBt2/assignment_loss‚0.13199835769178825 2loss/FBt3/assignment_loss‚0.052113621363878336 /loss/FRt2/detection_loss‚1.7036469058012185 /loss/FBt1/detection_loss‚2.0287289809096944 % _timestamp‚1.7885228375143435e+09ï»FGm“Á+‚(:ˆ ×âêÔé¤Ýÿ gpu.0.gpu‚48.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚163.874' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚40.9685 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¿ ÷îoÂ+‚(:ä ×âêÔÿ„´ disk.rbd0.out‚ 3.591796875 disk.md0.out‚ 1375.484375 disk.md0p1.out‚ 1375.484375" disk.sda2.out‚1426.71435546875$ memory_percent‚91.60161018371582 disk.nvme1n1.in‚ 498.80078125 disk.sda1.in‚0 disk.sda1.out‚0! disk.dm-0.out‚139485.59765625 proc.cpu.threads‚17 disk.rbd2.out‚ 3.529296875" disk.sdb2.out‚1426.71435546875 proc.memory.rssMB‚ 2912.78125 cpu‚25.20181549704771 network.sent‚ 2.8027761e+07 network.recv‚ 4.010876e+06 disk.rbd0.in‚0 disk.sdb2.in‚ 7.0546875 disk.sda2.in‚ 10.7578125 disk.sdb1.in‚0 disk.sdb.in‚ 7.0546875 disk.sda.in‚ 10.7578125! disk.sda.out‚1426.71435546875) disk./.usagePercent‚75.88102367488368% disk./.usageGB‚2713.8729553222656 disk.md0p1.in‚17.8125 disk.nvme0n1.in‚ 619.63671875" disk.nvme0n1.out‚ 73221.9765625( proc.memory.availableMB‚ 1375.9921875) proc.memory.percent‚17.77820587158203 disk.md0.in‚17.8125 disk.rbd2.in‚0# disk.nvme1n1.out‚66263.62109375 disk.sdb1.out‚0 disk.dm-0.in‚ 1118.4375! disk.sdb.out‚1426.71435546875Qs¥“Ã+‚(:ˆ æâêÔñŸ¤ÿ gpu.0.gpu‚35.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚163.422' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚40.8555 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¡[l™kÄ+‚(:à æâêÔ¸°€ disk.sda.in‚ 10.7578125 disk.sda.out‚1428.2333984375! disk.sda2.out‚1428.2333984375 disk.rbd0.in‚0 proc.memory.rssMB‚ 2912.78125 disk.md0.out‚ 1376.9453125 disk.dm-0.out‚140032.4140625 network.recv‚ 4.016737e+06$ disk./.usageGB‚2713.872917175293 disk.md0.in‚17.8125 cpu‚25.201617762292948 disk.md0p1.out‚ 1376.9453125 disk.nvme0n1.in‚ 619.63671875 network.sent‚ 2.8124997e+07 disk.sda1.out‚0 disk.sda2.in‚ 10.7578125 disk.rbd2.out‚ 3.529296875 disk.sdb1.out‚0! disk.sdb2.out‚1428.2333984375 disk.md0p1.in‚17.8125 disk.rbd0.out‚ 3.591796875 disk.sda1.in‚0 disk.nvme1n1.in‚ 498.80078125 disk.sdb1.in‚0$ memory_percent‚91.60311222076416 proc.cpu.threads‚17 disk.rbd2.in‚0" disk.nvme1n1.out‚ 66263.8046875! disk.nvme0n1.out‚ 73768.609375) disk./.usagePercent‚75.88102260827836 disk.sdb2.in‚ 7.0546875 disk.dm-0.in‚ 1118.4375 disk.sdb.in‚ 7.0546875) proc.memory.availableMB‚ 1375.74609375) proc.memory.percent‚17.77820587158203 disk.sdb.out‚1428.2333984375.¡Æ”Å+‚(:‰ õâêÔø¨ºÿ gpu.0.gpu‚35.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚164.989' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚41.24725 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚04#ïVÆ+‚(:Ë õâêÔôœØ€ disk.sda1.out‚0 disk.sdb1.in‚0! disk.dm-0.out‚140033.64453125) proc.memory.availableMB‚ 1376.24609375 network.sent‚ 2.8130295e+07% disk./.usageGB‚2713.8729400634766 disk.sdb1.out‚0 disk.md0.in‚17.8125) proc.memory.percent‚17.77820587158203 cpu‚25.20141482283017 disk.sda2.out‚ 1430.1875 disk.md0.out‚ 1378.8515625 disk.sdb2.in‚ 7.0546875 disk.sdb.in‚ 7.0546875 disk.sda.in‚ 10.7578125" disk.nvme1n1.out‚ 66264.5546875 disk.dm-0.in‚ 1118.4375 disk.md0p1.out‚ 1378.8515625 disk.nvme0n1.in‚ 619.63671875 disk.sdb.out‚ 1430.1875 proc.cpu.threads‚17 disk.rbd0.in‚0 disk.sda1.in‚0# disk.nvme0n1.out‚73769.08984375 proc.memory.rssMB‚ 2912.78125 disk.rbd0.out‚ 3.591796875 disk.rbd2.in‚0 disk.sdb2.out‚ 1430.1875 disk.md0p1.in‚17.8125$ memory_percent‚91.60006046295166 network.recv‚ 4.020435e+06) disk./.usagePercent‚75.88102324824155 disk.sda.out‚ 1430.1875 disk.sda2.in‚ 10.7578125 disk.nvme1n1.in‚ 498.80078125 disk.rbd2.out‚ 3.529296875½~:”Ç+‚(:‰ „ãêÔÜïæÿ gpu.0.gpu‚44.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚36.0 gpu.0.powerWatts‚114.131' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.53275 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0î&ktÈ+‚(:é „ãêÔÔ­Õ€ network.recv‚ 4.024689e+06 disk.sdb2.in‚ 7.0546875 disk.sda1.in‚0 disk.md0p1.in‚17.8125! disk.sda.out‚1431.61474609375 disk.rbd0.out‚ 3.591796875 network.sent‚ 2.8136215e+07# disk.nvme0n1.out‚74299.19921875 disk.sdb1.out‚0 disk.md0.in‚17.8125 disk.md0.out‚ 1380.23046875 disk.dm-0.in‚ 1118.4375 disk.sdb.in‚ 7.0546875" disk.sda2.out‚1431.61474609375% disk./.usageGB‚2713.3431129455566 disk.md0p1.out‚ 1380.23046875 disk.nvme0n1.in‚ 619.63671875 disk.sda.in‚ 10.7578125 disk.rbd0.in‚0 disk.rbd2.in‚0 proc.memory.rssMB‚ 2912.78125) proc.memory.percent‚17.77820587158203! disk.dm-0.out‚140563.93359375 cpu‚25.201213789798317 proc.cpu.threads‚17 disk.nvme1n1.in‚ 498.80078125! disk.nvme1n1.out‚ 66264.734375 disk.sdb1.in‚0 disk.rbd2.out‚ 3.529296875) disk./.usagePercent‚75.86620906027464" disk.sdb2.out‚1431.61474609375! disk.sdb.out‚1431.61474609375$ memory_percent‚91.59872531890869 disk.sda1.out‚0 disk.sda2.in‚ 10.7578125) proc.memory.availableMB‚ 1376.46484375øŒ*W4É+/ _wandbruntime‚12520  _runtime‚12520ˆpÐ ÉÊ+à _runtime‚12520.490311188 _step‚880 & lr-AdamW/pg1‚3.174412220384091e-05 & lr-AdamW/pg2‚3.174412220384091e-05 trainer/global_step‚10749 % _timestamp‚1.7885228950198298e+09YŽÿÎË+È _runtime‚12520.490311188 _step‚880 & lr-AdamW/pg1‚3.174412220384091e-05 & lr-AdamW/pg2‚3.174412220384091e-05 trainer/global_step‚10749 % _timestamp‚1.7885228950198298e+09𥽠4Ì+/ _wandbruntime‚12520  _runtime‚12520PêÄà (Í+„P QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 221613260 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12211748082688 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚617 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 221613260 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚863391697232816 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955591168 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407872 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 202251074 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12212647490560 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚863391714272176 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 137805815 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚64445878 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚895262752098816 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚883051022019072 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚988148480006144 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 222568448 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26090 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 246508544 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12211748082688 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12166220779455 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 trainer/global_step‚10749 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚883051004016128 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26088 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1000903959077888 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12167118190875 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1212 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407872 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚883051022019072 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚875557918012271 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚883051004016128 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12212647490560 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚619 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12755503011840 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚988148456066048 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917410816 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 491114478 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚64445261 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚875558832463051 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007241 % _timestamp‚1.7885228950213609e+09 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚895263669509632 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1187 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214665 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 _step‚881 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12755725580288 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 137805813 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚895263669509632 _runtime‚12520.492183667 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177047040 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 269527308 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728áŸêF"Î+‰P =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚895262752098816 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚883051022019072 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚988148480006144 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 222568448 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26090 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 246508544 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12211748082688 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12166220779455 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 trainer/global_step‚10749 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚883051004016128 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26088 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1000903959077888 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12167118190875 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1212 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407872 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚883051022019072 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚875557918012271 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚883051004016128 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12212647490560 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚619 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12755503011840 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚988148456066048 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917410816 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 491114478 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚64445261 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚875558832463051 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007241 % _timestamp‚1.7885228950213609e+09 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚895263669509632 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1187 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214665 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 _step‚881 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12755725580288 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 137805813 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚895263669509632 _runtime‚12520.492183667 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177047040 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 269527308 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 221613260 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12211748082688 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚617 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 221613260 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚863391697232816 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955591168 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407872 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 202251074 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12212647490560 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚863391714272176 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 137805815 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚64445878 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 J8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26114 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 491185083 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 269551925 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 202269917 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚875637006767717 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955591168 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177047040 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917424128 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚863469694366304 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 202269279 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12168226852401 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 269551925 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899421184 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚883130652314624 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 221633156 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 trainer/global_step‚10749 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚64451726 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 269525811 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1000994428061184 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 491185083 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚883130634311680 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚988237775286784 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 137818191 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450988 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1000994685042176 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚875637921218705 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411628 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12213760307200 _runtime‚12521.671606207 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 137818191 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450988 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1000994685042176 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚875637921218705 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411628 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12213760307200 _runtime‚12521.671606207 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26114 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 491185083 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 269551925 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 202269917 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚875637006767717 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955591168 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177047040 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917424128 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚863469694366304 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 202269279 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12168226852401 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 269551925 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899421184 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚883130652314624 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 221633156 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 trainer/global_step‚10749 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚64451726 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 269525811 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0ò"¯sÈ4Ò+/ _wandbruntime‚12521  _runtime‚12521AHØZ]Ó+× 0loss/FRt3/assignment_loss‚2.6625331044197083 1loss/SRqqt1/assignment_loss‚1.381395408810646 .loss/FRt2/detection_loss‚1.508659737222657 0loss/FRt1/assignment_loss‚2.7629778724090728 1loss/FBt3/assignment_loss‚0.32368188122442615 /loss/FRt1/detection_loss‚1.7593384305187547 1loss/SRqqt1/detection_loss‚1.8112822645435147 2loss/SRqqt3/assignment_loss‚1.1418242112404722 2loss/FBt1/assignment_loss‚0.018402026712364104 %loss/total_loss‚38.04419447691348 0loss/FRt2/assignment_loss‚0.9269218384003153 /loss/FRt3/detection_loss‚2.3679951252632363 /loss/FRt4/detection_loss‚3.1133779975084157 /loss/FBt2/detection_loss‚0.5324882336775772 trainer/global_step‚10749 _runtime‚12521.673823038 2loss/SRqqt2/assignment_loss‚1.0526652001745758 .loss/FBt1/detection_loss‚2.355210860570272 % _timestamp‚1.7885228962034748e+09 /loss/FRt4/assignment_loss‚2.656863217170422 2loss/SRqqt4/assignment_loss‚1.4009437096063737 0loss/FBt2/assignment_loss‚0.0877114392466316 0loss/FBt4/assignment_loss‚0.1608932281081555 0loss/SRqqt2/detection_loss‚0.899754433562198 1loss/SRqqt3/detection_loss‚1.4639503200921942 0loss/SRqqt4/detection_loss‚2.715853084016729 .loss/FBt3/detection_loss‚2.567469591895739 /loss/FBt4/detection_loss‚2.3720012605190277 epoch‚8 _step‚883ÛH hbÔ+Ü 0loss/FRt3/assignment_loss‚2.6625331044197083 1loss/SRqqt1/assignment_loss‚1.381395408810646 .loss/FRt2/detection_loss‚1.508659737222657 0loss/FRt1/assignment_loss‚2.7629778724090728 1loss/FBt3/assignment_loss‚0.32368188122442615 /loss/FRt1/detection_loss‚1.7593384305187547 1loss/SRqqt1/detection_loss‚1.8112822645435147 2loss/SRqqt3/assignment_loss‚1.1418242112404722 2loss/FBt1/assignment_loss‚0.018402026712364104 %loss/total_loss‚38.04419447691348 0loss/FRt2/assignment_loss‚0.9269218384003153 /loss/FRt3/detection_loss‚2.3679951252632363 /loss/FRt4/detection_loss‚3.1133779975084157 /loss/FBt2/detection_loss‚0.5324882336775772 trainer/global_step‚10749 _runtime‚12521.673823038 2loss/SRqqt2/assignment_loss‚1.0526652001745758 .loss/FBt1/detection_loss‚2.355210860570272 % _timestamp‚1.7885228962034748e+09 /loss/FRt4/assignment_loss‚2.656863217170422 2loss/SRqqt4/assignment_loss‚1.4009437096063737 0loss/FBt2/assignment_loss‚0.0877114392466316 0loss/FBt4/assignment_loss‚0.1608932281081555 0loss/SRqqt2/detection_loss‚0.899754433562198 1loss/SRqqt3/detection_loss‚1.4639503200921942 0loss/SRqqt4/detection_loss‚2.715853084016729 .loss/FBt3/detection_loss‚2.567469591895739 /loss/FBt4/detection_loss‚2.3720012605190277 epoch‚8 _step‚883óµ1郞Õ+‚(:“ “ãêÔ¦¯à€ gpu.0.gpu‚38.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚37.0 gpu.0.powerWatts‚115.634' gpu.0.enforcedPowerLimitWatts‚400.0) gpu.0.powerPercent‚28.908499999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0WÉW!bÖ+‚(:× “ãêÔôû€ disk.rbd0.in‚0 disk.sdb1.out‚0 disk.md0.out‚ 1382.21875 disk.rbd2.in‚0 disk.sda1.out‚0! disk.dm-0.out‚140752.77734375 cpu‚25.201011900279852 disk.sda2.in‚ 10.7578125 disk.rbd0.out‚ 3.591796875 disk.sda1.in‚0) proc.memory.percent‚17.77820587158203 network.recv‚ 4.028959e+06' disk./.usagePercent‚75.866200314111# disk.nvme0n1.out‚74487.92578125 disk.rbd2.out‚ 3.529296875 disk.sdb2.in‚ 7.0546875 disk.md0p1.out‚ 1382.21875 disk.nvme1n1.in‚ 498.80078125" disk.nvme1n1.out‚ 66264.8515625 disk.sda2.out‚1433.669921875 disk.md0.in‚17.8125 disk.nvme0n1.in‚ 619.63671875 disk.sdb.out‚1433.669921875 disk.sda.out‚1433.669921875 disk.md0p1.in‚17.8125' proc.memory.availableMB‚ 1376.203125 network.sent‚ 2.8140223e+07 disk.sda.in‚ 10.7578125 disk.sdb1.in‚0 disk.dm-0.in‚ 1118.4375 proc.memory.rssMB‚ 2912.78125 proc.cpu.threads‚17 disk.sdb2.out‚1433.669921875$ memory_percent‚91.60032272338867$ disk./.usageGB‚2713.342800140381 disk.sdb.in‚ 7.0546875ÕN”×+‚(:‰ ¢ãêÔÛþ ÿ gpu.0.gpu‚47.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚37.0 gpu.0.powerWatts‚174.903' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.72575 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 gˆnØ+‚(:ã ¢ãêÔï‚Åþ disk.rbd2.in‚0" disk.sdb2.out‚1435.26513671875 network.sent‚ 2.824807e+07% disk./.usageGB‚2713.2803115844727 disk.dm-0.in‚ 1118.4375) disk./.usagePercent‚75.86445310793431# disk.nvme0n1.out‚74622.70703125 disk.sdb.in‚ 7.0546875 disk.rbd0.out‚ 3.591796875 disk.rbd2.out‚ 3.529296875 cpu‚25.200801994224395 disk.md0p1.out‚ 1383.765625 disk.sdb1.in‚0 disk.md0.in‚17.8125 proc.memory.rssMB‚ 2912.78125 proc.cpu.threads‚17 disk.sda1.in‚0! disk.sda.out‚1435.26513671875 disk.sda2.in‚ 10.7578125" disk.nvme1n1.out‚ 66265.1640625 disk.sda1.out‚0 disk.md0p1.in‚17.8125" disk.sda2.out‚1435.26513671875 disk.rbd0.in‚0 disk.nvme1n1.in‚ 498.80078125 disk.sdb2.in‚ 7.0546875$ memory_percent‚91.60184860229492! disk.dm-0.out‚140887.87109375' proc.memory.availableMB‚ 1375.953125) proc.memory.percent‚17.77820587158203 network.recv‚ 4.034413e+06 disk.nvme0n1.in‚ 619.63671875 disk.sda.in‚ 10.7578125 disk.sdb1.out‚0 disk.md0.out‚ 1383.765625! disk.sdb.out‚1435.26513671875ßW„“Ù+‚(:ˆ ±ãêÔªÀêÿ gpu.0.gpu‚24.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚37.0 gpu.0.powerWatts‚134.298' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚33.5745 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Í&øoÚ+‚(:ä ±ãêÔ݃Ö disk.sda1.in‚0 disk.sda1.out‚0 network.recv‚ 4.037819e+06" disk.sda2.out‚1436.84521484375 disk.nvme1n1.in‚ 498.80078125 disk.rbd2.in‚0 disk.rbd2.out‚ 3.529296875 disk.sdb1.in‚0" disk.sdb2.out‚1436.84521484375 disk.dm-0.in‚ 1118.4375 disk.md0p1.out‚ 1385.28515625# disk.nvme0n1.out‚74884.93359375 disk.sdb1.out‚0) disk./.usagePercent‚75.86696048372343# disk.nvme1n1.out‚66357.22265625 disk.md0.out‚ 1385.28515625 disk.dm-0.out‚ 141242.15625 cpu‚25.20060905988028$ disk./.usageGB‚2713.369987487793$ memory_percent‚91.60032272338867) proc.memory.percent‚17.77820587158203 disk.sda2.in‚ 10.7578125 disk.nvme0n1.in‚ 619.63671875 proc.cpu.threads‚17! disk.sda.out‚1436.84521484375 disk.rbd0.out‚ 3.591796875 disk.sdb.in‚ 7.0546875! disk.sdb.out‚1436.84521484375' proc.memory.availableMB‚ 1376.203125 proc.memory.rssMB‚ 2912.78125 disk.md0p1.in‚17.8125 disk.sda.in‚ 10.7578125 disk.rbd0.in‚0 disk.md0.in‚17.8125 network.sent‚ 2.8255676e+07 disk.sdb2.in‚ 7.0546875¤”ib”Û+‚(:‰ ÀãêÔžñ£ÿ gpu.0.gpu‚50.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚37.0 gpu.0.powerWatts‚160.305' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚40.07625 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0h œrÜ+‚(:ç ÀãêÔ©«·ÿ! disk.dm-0.out‚141409.40234375 disk.md0p1.in‚17.8125 disk.sda2.in‚ 10.7578125 disk.rbd0.in‚0 disk.md0.in‚17.8125 proc.cpu.threads‚17$ disk./.usageGB‚2713.435432434082 disk.dm-0.in‚ 1118.4375 disk.nvme0n1.in‚ 619.63671875 disk.rbd0.out‚ 3.591796875 disk.nvme1n1.in‚ 498.80078125 disk.sdb1.in‚0 disk.sdb2.in‚ 7.0546875) proc.memory.percent‚17.77820587158203 network.sent‚ 2.8259517e+07# disk.nvme0n1.out‚75013.01171875 disk.sdb.in‚ 7.0546875) proc.memory.availableMB‚ 1376.18359375( disk./.usagePercent‚75.8687903518125 disk.md0p1.out‚ 1387.08984375! disk.sdb.out‚1438.72021484375" disk.sda2.out‚1438.72021484375 disk.md0.out‚ 1387.08984375 disk.rbd2.out‚ 3.529296875" disk.sdb2.out‚1438.72021484375$ memory_percent‚91.60044193267822 network.recv‚ 4.042328e+06 disk.rbd2.in‚0 proc.memory.rssMB‚ 2912.78125 disk.sda1.in‚0 disk.sda.in‚ 10.7578125! disk.sda.out‚1438.72021484375! disk.nvme1n1.out‚ 66396.390625 disk.sdb1.out‚0 cpu‚25.200422910751055 disk.sda1.out‚0~Í)4Ý+/ _wandbruntime‚12579  _runtime‚12579c$ŠËÞ+Å _runtime‚12579.032021105 _step‚884 ' lr-AdamW/pg1‚2.9448029404478835e-05 ' lr-AdamW/pg2‚2.9448029404478835e-05 trainer/global_step‚10799 % _timestamp‚1.7885229535614986e+09XÇK×Ðß+Ê ' lr-AdamW/pg2‚2.9448029404478835e-05 trainer/global_step‚10799 % _timestamp‚1.7885229535614986e+09 _runtime‚12579.032021105 _step‚884 ' lr-AdamW/pg1‚2.9448029404478835e-05ô‰Î´¤4à+/ _wandbruntime‚12579  _runtime‚12579ÁõLá+ƒP YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚879513283826559 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12268302070784 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177047040 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 226762752 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚887032518793728 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 401960960 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚992614441042944 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚992614417102848 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26090 QCDeviceStatsMonitor.on_train_ba…ùZ)½!tch_start/allocation.large_pool.freed‚ 222608158 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1005428336246272 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚879512369375779 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 203159552 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214665 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955591168 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚887032536796672 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12267402662912 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12268302070784 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 $ _timestamp‚1.788522953563068e+09 _runtime‚12579.033827754 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 493342280 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 270760210 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12813919143424 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 222608160 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26088 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 138424613 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007241 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚887032536796672 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚887032518793728 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12222564847983 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407872 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 250702848 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917410816 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1187 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚867290701939216 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411420 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚899299921456640 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 270734122 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450780 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 138424615 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1212 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12267402662912 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955591168 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚867290718978576 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 222608158 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 270760210 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚617 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _step‚885 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090FdL(â+ˆP H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1212 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12267402662912 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955591168 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚867290718978576 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 222608158 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 270760210 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚617 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _step‚885 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚879513283826559 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12268302070784 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177047040 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 226762752 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚887032518793728 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 401960960 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚992614441042944 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚992614417102848 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26090 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 222608158 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1005428336246272 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚879512369375779 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 203159552 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214665 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955591168 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚887032536796672 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12267402662912 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12268302070784 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 $ _timestamp‚1.788522953563068e+09 _runtime‚12579.033827754 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 493342280 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 270760210 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12813919143424 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 222608160 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26088 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 138424613 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007241 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚887032536796672 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚887032518793728 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12222564847983 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407872 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 250702848 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917410816 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1187 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚867290701939216 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411420 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚899299921456640 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 270734122 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450780 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 138424615 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728õ¾êp¹4ã+/ _wandbruntime‚12580  _runtime‚12580U×Ú'ä+ŽN 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411628 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 493386769 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 _runtime‚12580.031522007 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚899381581979648 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 236009472 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917424128 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 203177700 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 270758713 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955591168 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚887112149089280 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚887112167092224 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007241 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚899381581979648 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 138436991 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 401960960 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 493386769 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12269414887424 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 493412885 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚992703736323584 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 222628056 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚879592372582213 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177047040 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 270784827 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚64741329 % _timestamp‚1.7885229545605457e+09 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 493412885 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899421184 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚867368699072704 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12268515466240 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31686 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12222776097881 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚992703760263680 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12815108776448 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚64740711 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚879591458131225 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚887112149089280 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚620 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚899380664555520 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 222628056 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚618 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 212069376 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚867368682033344 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 270758713 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12815320845824 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26114 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚887112167092224 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12268515466240 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177047040 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 222628058 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899421184 trainer/global_step‚10799 _step‚886 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1187 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1005518845100032 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0ŠÊå+“N REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚899380664555520 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 222628056 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚618 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 212069376 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚867368682033344 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 270758713 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12815320845824 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26114 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚887112167092224 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12268515466240 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177047040 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 222628058 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899421184 trainer/global_step‚10799 _step‚886 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1187 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1005518845100032 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411628 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 493386769 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 _runtime‚12580.031522007 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚899381581979648 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 236009472 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917424128 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 203177700 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 270758713 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955591168 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚887112149089280 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚887112167092224 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007241 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚899381581979648 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 138436991 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 401960960 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 493386769 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12269414887424 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 493412885 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚992703736323584 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 222628056 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚879592372582213 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177047040 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 270784827 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚64741329 % _timestamp‚1.7885229545605457e+09 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 493412885 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899421184 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚867368699072704 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12268515466240 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31686 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12222776097881 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚992703760263680 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12815108776448 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚64740711 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚879591458131225 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚887112149089280 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚620öãÛWí4æ+/ _wandbruntime‚12580  _runtime‚12580Ðá¡`ç+Ú 1loss/FBt2/assignment_loss‚0.08791357500310915 /loss/FRt3/detection_loss‚2.1733772984809345 0loss/FRt1/assignment_loss‚2.0918343921877303 .loss/FRt3/assignment_loss‚2.32970728609297 1loss/FBt4/assignment_loss‚0.04964671864013326 1loss/SRqqt2/detection_loss‚0.9654567552996534 % _timestamp‚1.7885229545629025e+09 2loss/SRqqt2/assignment_loss‚0.9263780582430107 /loss/FRt2/detection_loss‚1.4425755561888218 /loss/FRt4/detection_loss‚3.1729814410209656 _step‚887 /loss/FRt4/assignment_loss‚2.800314290523529 2loss/FBt3/assignment_loss‚0.008117444240026219 /loss/FRt1/detection_loss‚1.7844157847667823 _runtime‚12580.033006884 1loss/SRqqt3/detection_loss‚1.8381721249648504 1loss/SRqqt4/detection_loss‚2.1835316859185694 .loss/FBt3/detection_loss‚2.899837281022753 0loss/FRt2/assignment_loss‚1.5370168173313141 2loss/FBt1/assignment_loss‚0.004115338524873853 /loss/FBt2/detection_loss‚0.6515907124910307 trainer/global_step‚10799 1loss/SRqqt1/assignment_loss‚1.122192802518839 2loss/SRqqt3/assignment_loss‚1.3200895121435121 .loss/FBt4/detection_loss‚3.274126172065735 epoch‚8 2loss/SRqqt4/assignment_loss‚1.8457582123577594 1loss/SRqqt1/detection_loss‚1.2979515448096208 .loss/FBt1/detection_loss‚2.301541710893313 %loss/total_loss‚38.10864251572984äµ¾eè+ß 2loss/SRqqt4/assignment_loss‚1.8457582123577594 1loss/SRqqt1/detection_loss‚1.2979515448096208 .loss/FBt1/detection_loss‚2.301541710893313 %loss/total_loss‚38.10864251572984 1loss/FBt2/assignment_loss‚0.08791357500310915 /loss/FRt3/detection_loss‚2.1733772984809345 0loss/FRt1/assignment_loss‚2.0918343921877303 .loss/FRt3/assignment_loss‚2.32970728609297 1loss/FBt4/assignment_loss‚0.04964671864013326 1loss/SRqqt2/detection_loss‚0.9654567552996534 % _timestamp‚1.7885229545629025e+09 2loss/SRqqt2/assignment_loss‚0.9263780582430107 /loss/FRt2/detection_loss‚1.4425755561888218 /loss/FRt4/detection_loss‚3.1729814410209656 _step‚887 /loss/FRt4/assignment_loss‚2.800314290523529 2loss/FBt3/assignment_loss‚0.008117444240026219 /loss/FRt1/detection_loss‚1.7844157847667823 _runtime‚12580.033006884 1loss/SRqqt3/detection_loss‚1.8381721249648504 1loss/SRqqt4/detection_loss‚2.1835316859185694 .loss/FBt3/detection_loss‚2.899837281022753 0loss/FRt2/assignment_loss‚1.5370168173313141 2loss/FBt1/assignment_loss‚0.004115338524873853 /loss/FBt2/detection_loss‚0.6515907124910307 trainer/global_step‚10799 1loss/SRqqt1/assignment_loss‚1.122192802518839 2loss/SRqqt3/assignment_loss‚1.3200895121435121 .loss/FBt4/detection_loss‚3.274126172065735 epoch‚8÷àǾ’é+‚(:‡ ÏãêÔ—ª¸þ gpu.0.gpu‚56.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚175.844' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.961 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0R«iê+‚(:Þ ÏãêÔ³éÎÿ disk.dm-0.in‚ 1118.4375 disk.dm-0.out‚141563.5859375 disk.rbd0.out‚ 3.591796875 disk.sdb1.out‚0 disk.md0p1.out‚ 1388.6640625 disk.sda.in‚ 10.7578125 proc.memory.rssMB‚ 2912.78125) proc.memory.percent‚17.77820587158203 proc.cpu.threads‚17 disk.md0.in‚17.8125 disk.md0.out‚ 1388.6640625 disk.rbd2.in‚0! disk.sdb2.out‚1440.3681640625 disk.sda1.in‚0 disk.sda2.in‚ 10.7578125$ memory_percent‚91.60234928131104) proc.memory.availableMB‚ 1375.87109375 network.recv‚ 4.04573e+06 disk.nvme1n1.in‚ 498.80078125 network.sent‚ 2.8277997e+07) disk./.usagePercent‚75.87405383575165$ disk./.usageGB‚2713.623680114746 disk.sdb1.in‚0 disk.sdb.out‚1440.3681640625 cpu‚25.20019987497316 disk.rbd0.in‚0! disk.nvme1n1.out‚ 66468.546875 disk.rbd2.out‚ 3.529296875 disk.sdb2.in‚ 7.0546875 disk.md0p1.in‚17.8125" disk.nvme0n1.out‚ 75095.0390625! disk.sda2.out‚1440.3681640625 disk.sda.out‚1440.3681640625 disk.sda1.out‚0 disk.nvme0n1.in‚ 619.63671875 disk.sdb.in‚ 7.0546875úËj“ë+‚(:ˆ ÞãêԢĥÿ gpu.0.gpu‚42.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚165.266' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚41.3165 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0C¼oì+‚(:ä Þãêԣ䷀ disk.sdb.in‚ 7.0546875! disk.sda.out‚1441.92822265625 disk.md0.in‚17.8125# memory_percent‚91.6008472442627 disk.rbd0.in‚0 disk.nvme1n1.in‚ 498.80078125 disk.md0.out‚ 1390.16796875( proc.memory.availableMB‚ 1376.1171875 network.sent‚ 2.8374636e+07% disk./.usageGB‚2713.4827270507812 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-0.out‚141864.7578125! disk.sdb.out‚1441.92822265625 disk.sda2.in‚ 10.7578125 disk.rbd2.out‚ 3.529296875 disk.nvme0n1.out‚ 75252.28125 disk.sdb1.out‚0 proc.cpu.threads‚17 disk.md0p1.in‚17.8125" disk.sda2.out‚1441.92822265625" disk.nvme1n1.out‚ 66612.4765625" disk.sdb2.out‚1441.92822265625) proc.memory.percent‚17.77820587158203 cpu‚25.200017990027117 network.recv‚ 4.051893e+06 disk.dm-0.in‚ 1118.4375 disk.rbd0.out‚ 3.591796875 disk.rbd2.in‚0 proc.memory.rssMB‚ 2912.78125 disk.nvme0n1.in‚ 619.63671875 disk.sda.in‚ 10.7578125 disk.sdb1.in‚0 disk.sdb2.in‚ 7.0546875) disk./.usagePercent‚75.87011272908975 disk.md0p1.out‚ 1390.16796875¿Ð=M”í+‚(:‰ íãêÔ÷Ü“€ gpu.0.gpu‚55.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚172.395' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.09875 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0G•ídî+‚(:Ù íãêÔõÙ®€ disk.rbd0.out‚ 3.591796875 disk.nvme1n1.in‚ 498.80078125 disk.md0.out‚1391.625 disk.rbd2.out‚ 3.529296875" disk.sdb2.out‚1443.44482421875 disk.md0p1.out‚1391.625 disk.sda.in‚ 10.7578125 cpu‚25.19978140480589 disk.rbd0.in‚0 disk.sdb1.out‚0 disk.sdb2.in‚ 7.0546875! disk.sdb.out‚1443.44482421875 network.recv‚ 4.055611e+06) disk./.usagePercent‚75.85155688965601 disk.sda1.out‚0! disk.dm-0.out‚142061.66015625 disk.sdb.in‚ 7.0546875) proc.memory.availableMB‚ 1376.12109375 proc.memory.rssMB‚ 2912.78125 disk.rbd2.in‚0) proc.memory.percent‚17.77820587158203 network.sent‚ 2.837769e+07 disk.sda1.in‚0 disk.md0p1.in‚17.8125$ disk./.usageGB‚2712.819080352783 disk.md0.in‚17.8125 disk.dm-0.in‚ 1118.4375 disk.nvme0n1.in‚ 619.63671875" disk.sda2.out‚1443.44482421875 disk.nvme1n1.out‚ 66720.875! disk.sda.out‚1443.44482421875 disk.sda2.in‚ 10.7578125$ memory_percent‚91.60082340240479 proc.cpu.threads‚17 disk.sdb1.in‚0# disk.nvme0n1.out‚75340.78515625rié.’ï+‚(:‡ üãêÔò¾”ÿ gpu.0.gpu‚17.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚175.512' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.878 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0_aÀkð+‚(:à üãêÔöþ…€ disk.md0p1.out‚ 1393.72265625! disk.sda.out‚1445.59619140625 cpu‚25.199584979243166 disk.md0.in‚17.8125 disk.dm-0.in‚ 1118.4375" disk.nvme0n1.out‚ 75340.7890625 disk.rbd0.out‚ 3.591796875 proc.cpu.threads‚17) disk./.usagePercent‚75.85155806292187 disk.md0.out‚ 1393.72265625 disk.rbd2.out‚ 3.529296875) proc.memory.percent‚17.77820587158203 disk.sdb1.in‚0 disk.sdb2.in‚ 7.0546875 disk.nvme0n1.in‚ 619.63671875 disk.sda2.in‚ 10.7578125 disk.rbd0.in‚0$ disk./.usageGB‚2712.819122314453 disk.nvme1n1.in‚ 498.80078125" disk.sda2.out‚1445.59619140625$ memory_percent‚91.60079956054688$ proc.memory.availableMB‚1376.125" disk.nvme1n1.out‚ 66792.9296875 disk.sdb1.out‚0" disk.sdb2.out‚1445.59619140625 disk.sda1.in‚0 disk.sda1.out‚0 disk.sdb.in‚ 7.0546875! disk.sdb.out‚1445.59619140625 disk.sda.in‚ 10.7578125 network.sent‚ 2.8385891e+07 network.recv‚ 4.060017e+06 disk.rbd2.in‚0 disk.md0p1.in‚17.8125 proc.memory.rssMB‚ 2912.78125 disk.dm-0.out‚ 142133.71875œ¿.þ4ñ+/ _wandbruntime‚12632  _runtime‚12632ºGþÎÊò+Ä ' lr-AdamW/pg2‚2.7235171439712757e-05 trainer/global_step‚10849 % _timestamp‚1.7885230065351694e+09 _runtime‚12632.00548278 _step‚888 ' lr-AdamW/pg1‚2.7235171439712757e-054RÓÏó+É ' lr-AdamW/pg1‚2.7235171439712757e-05 ' lr-AdamW/pg2‚2.7235171439712757e-05 trainer/global_step‚10849 % _timestamp‚1.7885230065351694e+09 _runtime‚12632.00548278 _step‚888øÁŽ ª4ô+/ _wandbruntime‚12632  _runtime‚12632îú€ (õ+„P JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12872608590336 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚883467735190067 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚871189723684976 _runtime‚12632.006601135 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 139043415 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 495596172 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 271993112 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1212 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 271967024 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 495570082 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 246508544 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214665 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 204069479 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚903338008225280 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚903338008225280 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407872 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1187 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007241 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚891014033571328 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚883466820739287 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411420 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955591168 trainer/global_step‚10849 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12323956651008 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1009953010670080 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 222568448 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚9033370908144645 I=ö+‰P ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955591168 trainer/global_step‚10849 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12323956651008 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1009953010670080 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 222568448 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚903337090814464 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12872608590336 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚883467735190067 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚871189723684976 _runtime‚12632.006601135 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 139043415 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 495596172 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 271993112 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1212 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 271967024 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 495570082 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 246508544 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214665 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 204069479 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚903338008225280 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚903338008225280 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407872 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1187 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007241 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚891014033571328 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚883466820739287 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411420 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607ù5œ;/4÷+/ _wandbruntime‚12632  _runtime‚12632(±/õ'ø+ŽN ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450988 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12278222754989 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 223622956 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚10849 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177047040 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚891093681869824 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 495614571 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 139055791 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 401960960 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 139055789 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26114 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚903418751337472 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12325069467648 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 248592384 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 % _timestamp‚1.7885230074959335e+09 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 223622958 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚891093663866880 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177047040 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚997169697360384 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 224652288 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 223622958 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 272017729 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚891093681869824 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 271991615 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12325069467648 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899421184 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 495640687 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 204086958 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚903418751337472 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 223622956 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚883545909494733 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411628 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚903417833913344 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 495640687 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1010043247994880 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚871267686739744 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚891093663866880 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚65031169 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12324170046464 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚626 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12873550634496 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007241 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 495614571 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12324170046464 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚628 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚997169721300480 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 204087586 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1187 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899421184 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚883546823945721 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944Ä i'ù+“N MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26114 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 271991615 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚65031795 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚871267703779104 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 _step‚890 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177047040 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚997169697360384 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 224652288 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 223622958 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 272017729 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚891093681869824 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 271991615 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12325069467648 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899421184 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 495640687 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 204086958 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚903418751337472 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 223622956 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚883545909494733 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411628 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚903417833913344 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 495640687 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1010043247994880 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚871267686739744 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚891093663866880 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚65031169 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12324170046464 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚626 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12873550634496 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007241 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 495614571 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12324170046464 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚628 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚997169721300480 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 204087586 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1187 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899421184 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚883546823945721 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450988 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12278222754989 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 223622956 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚10849 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177047040 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚891093681869824 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 495614571 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 139055791 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 401960960 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 139055789 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26114 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚903418751337472 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12325069467648 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 248592384 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 % _timestamp‚1.7885230074959335e+09 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 223622958 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚891093663866880 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 FDeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_validation_batch_start/num_sync_all_streams‚2 UEDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.peak‚ 13955591168 \LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 SEDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.freed‚ 205366501 TKDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.peak‚1187 `MDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.freed‚12336696922475 OGDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.current‚564 FDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 _step‚892 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.freed‚65464496 eRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.freed‚12934025127936 B;DeviceStatsMonitor.on_validation_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_validation_batch_start/oversize_segments.freed‚0 YIDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.allocated‚ 14220787712 ]NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.current‚ 1182793728 QIDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.allocated‚607 VHDeviceStatsMonitor.on_validation_batch_start/active.small_pool.allocated‚ 273210691 ]JDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.freed‚12382851762176 NHDeviceStatsMonitor.on_validation_batch_start/oversize_segments.allocated‚0 F=DeviceStatsMonitor.on_validation_batch_start/num_device_alloc‚1171 LDDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.peak‚564 VHDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.freed‚ 273190172 iVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.allocated‚12934174192128 ^NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.current‚ 13037993984 trainer/global_step‚10880 LFDeviceStatsMonitor.on_validation_batch_start/active.large_pool.current‚2 RDDeviceStatsMonitor.on_validation_batch_start/active.large_pool.freed‚ 224988318 ZFDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.freed‚908570047304192 VNDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.current‚209 eQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.allocated‚876123821098448 OGDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.current‚607 ZLDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.freed‚ 139902005â%T»w*‹,ñT eQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.allocated‚896187213544960 WIDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.allocated‚ 205366712 F=DeviceStatsMonitor.on_validation_batch_start/segment.all.peak‚1171 K=DeviceStatsMonitor.on_validation_batch_start/active.all.freed‚ 498178490 VHDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.current‚ 693119488 aMDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.freed‚896187195542016 SEDeviceStatsMonitor.on_validation_batch_start/allocation.all.allocated‚ 498199011 ZLDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.current‚ 675116544 OIDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.current‚0 KBDeviceStatsMonitor.on_validation_batch_start/segment.all.allocated‚1171 VHDeviceStatsMonitor.on_validation_batch_start/active.large_pool.allocated‚ 224988320 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.freed‚0 KEDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.current‚ 693119488 WCDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.freed‚908570047304192 bNDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.allocated‚896187213544960 RKDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.peak‚35 LFDeviceStatsMonitor.on_validation_batch_start/oversize_segments.current‚0 `PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 aNDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.allocated‚12383526878720 ^PDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.allocated‚ 139902007 dODeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.allocated‚1015589815463424 gRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.freed‚1002655617331200 \ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.current‚17039360 PJDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.current‚2 aTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.current‚23940096 [LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.peak‚ 1175007241 [GDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.allocated‚908570740423680 VHDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.freed‚ 224988318 QGDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.peak‚31686 LFDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.peak‚0 \LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.peak‚ 12392107008 QIDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.allocated‚564 ZLDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.allocated‚ 224988320 OGDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.current‚211 MGDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.freed‚0 ZFDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.freed‚888460500981563 [KDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 YIDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.peak‚ 12835443712 ]ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.current‚ 675116544 `MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.freed‚12382851762176 _PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 OADeviceStatsMonitor.on_validation_batch_start/allocation.all.freed‚ 498178490 ZLDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.allocated‚ 273210691 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 bTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.current‚ 149064192 KEDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.freed‚0 MCDeviceStatsMonitor.on_validation_batch_start/allocation.all.current‚20521 TJDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.current‚20519 MDDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.peak‚1212 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.allocated‚65464705 QKDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.allocated‚0 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.freed‚0 MCDeviceStatsMonitor.on_validation_batch_start/active.small_pool.peak‚31686 RBDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.peak‚ 13955591168 UEDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.peak‚ 13510214665 aMDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.freed‚876123804059088 `KDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.freed‚1015589642459136 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 D>DeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_validation_batch_start/num_sync_all_streams‚2 UEDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.peak‚ 13955591168 \LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 SEDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.freed‚ 205366501 TKDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.peak‚1187 `MDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.freed‚12336696922475 OGDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.current‚564 FDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 _step‚892 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.freed‚65464496 eRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.freed‚12934025127936 B;DeviceStatsMonitor.on_validation_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_validation_batch_start/oversize_segments.freed‚0 YIDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.allocated‚ 14220787712 ]NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.current‚ 1182793728 QIDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.allocated‚607 VHDeviceStatsMonitor.on_validation_batch_start/active.small_pool.allocated‚ 273210691 ]JDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.freed‚12382851762176 NHDeviceStatsMonitor.on_validation_batch_start/oversize_segments.allocated‚0 F=DeviceStatsMonitor.on_validation_batch_start/num_device_alloc‚1171 LDDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.peak‚564 VHDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.freed‚ 273190172 iVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.allocated‚12934174192128 ^NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.current‚ 13037993984 trainer/global_step‚10880 LFDeviceStatsMonitor.on_validation_batch_start/active.large_pool.current‚2 RDDeviceStatsMonitor.on_validation_batch_start/active.large_pool.freed‚ 224988318 ZFDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.freed‚908570047304192 VNDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.current‚209 eQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.allocated‚876123821098448 OGDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.current‚607 ZLDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.freed‚ 139902005 dQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.allocated‚12337370067235 I@DeviceStatsMonitor.on_validation_batch_start/segment.all.current‚1171 OADeviceStatsMonitor.on_validation_batch_start/active.all.allocated‚ 498199011 ;5DeviceStatsMonitor.on_validation_batch_start/num_ooms‚0 ^JDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.allocated‚888461191165683 I?DeviceStatsMonitor.on_validation_batch_start/active.all.current‚20521 dQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.allocated‚12383526878720 [MDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.current‚ 173004288 TDDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.peak‚ 14220787712 _runtime‚12724.80407417 LCDeviceStatsMonitor.on_validation_batch_start/active.large_pool.peak‚3677 ^JDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.freed‚896187195542016 XIDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.peak‚ 1177047040 PGDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.peak‚3677 BDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚1171 KBDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.peak‚1212 JADeviceStatsMonitor.on_validation_batch_end/active.large_pool.peak‚3677 QCDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.current‚ 693119488 \HDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.freed‚896240766817792 RHDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.current‚20519 C=DeviceStatsMonitor.on_validation_batch_end/num_oom_rejections‚0 VGDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.peak‚ 1177047040 ePDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.freed‚1002711066803200 UADeviceStatsMonitor.on_validation_batch_end/active_bytes.all.freed‚908624107142656 SCDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.peak‚ 13955591168 H>DeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚31931 \HDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.allocated‚888511448182231 cODeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.allocated‚876173591188272 D;DeviceStatsMonitor.on_validation_batch_end/segment.all.peak‚1171 NDDeviceStatsMonitor.on_validation_batch_end/active.small_pool.current‚20519 PBDeviceStatsMonitor.on_validation_batch_end/active.small_pool.freed‚ 273198946 D:DeviceStatsMonitor.on_validation_batch_end/active.all.peak‚31931 NHDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.current‚2 NEDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.peak‚3677 XJDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.freed‚ 139911445 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.allocated‚65469916 [HDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.freed‚12383340324864 ^IDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.freed‚1015645580493824 JBDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.peak‚564 \HDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.allocated‚908624800262144 cODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.allocated‚896240784820736 E?DeviceStatsMonitor.on_validation_batch_end/num_sync_all_streams‚2 _LDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.allocated‚12384015441408 ]NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 MEDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.current‚211 JDDeviceStatsMonitor.on_validation_batch_end/oversize_segments.current‚0 ZMDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.current‚17039360 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.freed‚0 P@DeviceStatsMonitor.on_validation_batch_end/active_bytes.all.peak‚ 13955591168 TFDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.freed‚ 225003594 [MDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.current‚ 673144760 I@DeviceStatsMonitor.on_validation_batch_end/segment.all.allocated‚1171 G=DeviceStatsMonitor.on_validation_batch_end/active.all.current‚20521 WJDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_validation_batch_end/allocation.all.allocated‚ 498223061 93DeviceStatsMonitor.on_validation_batch_end/num_ooms‚0 trainer/global_step‚10880 ZMDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.current‚18002944 RLDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.current‚2 TLDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.current‚209 RIDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.peak‚1187 WGDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.allocated‚ 14220787712 _runtime‚12725.912323044 YEDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.allocated‚908624800262144 JDDeviceStatsMonitor.on_validation_batch_end/active.large_pool.current‚2 QCDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.freed‚ 205381152 YJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.peak‚ 1175007241 `LDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.allocated‚896240784820736 [MDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.current‚ 675116544 M?DeviceStatsMonitor.on_validation_batch_end/allocation.all.freed‚ 498202540 GADeviceStatsMonitor.on_validation_batch_end/oversize_segments.peak‚0 UEDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.current‚ 14220787712 ICDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.freed‚0 TFDeviceStatsMonitor.on_validation_batch_end/active.large_pool.allocated‚ 225003596 UGDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.allocated‚ 205381363 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.freed‚65469707 XJDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.allocated‚ 273219465 MGDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.current‚0 JDDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.peak‚0 OEDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.peak‚31686 ICDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.freed‚0 TFDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.current‚ 693119488 _KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.freed‚896240766817792 bODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.allocated‚12384015441408 ^NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 YIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 XIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 PIDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.peak‚35 XDDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.freed‚908624107142656 YJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.sm'$bJ› all_pool.peak‚ 1177047040 _RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.current‚23940096 KEDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.freed‚0 KADeviceStatsMonitor.on_validation_batch_end/allocation.all.current‚20521 ICDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.freed‚0 MEDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.current‚607 _step‚893 BDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚1171 KBDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.peak‚1212 JADeviceStatsMonitor.on_validation_batch_end/active.large_pool.peak‚3677 QCDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.current‚ 693119488 \HDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.freed‚896240766817792 RHDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.current‚20519 C=DeviceStatsMonitor.on_validation_batch_end/num_oom_rejections‚0 VGDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.peak‚ 1177047040 ePDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.freed‚1002711066803200 UADeviceStatsMonitor.on_validation_batch_end/active_bytes.all.freed‚908624107142656 SCDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.peak‚ 13955591168 H>DeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚31931 \HDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.allocated‚888511448182231 cODeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.allocated‚876173591188272 D;DeviceStatsMonitor.on_validation_batch_end/segment.all.peak‚1171 NDDeviceStatsMonitor.on_validation_batch_end/active.small_pool.current‚20519 PBDeviceStatsMonitor.on_validation_batch_end/active.small_pool.freed‚ 273198946 D:DeviceStatsMonitor.on_validation_batch_end/active.all.peak‚31931 NHDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.current‚2 NEDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.peak‚3677 XJDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.freed‚ 139911445 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.allocated‚65469916 [HDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.freed‚12383340324864 ^IDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.freed‚1015645580493824 JBDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.peak‚564 \HDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.allocated‚908624800262144 cODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.allocated‚896240784820736 E?DeviceStatsMonitor.on_validation_batch_end/num_sync_all_streams‚2 _LDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.allocated‚12384015441408 ]NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 MEDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.current‚211 JDDeviceStatsMonitor.on_validation_batch_end/oversize_segments.current‚0 ZMDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.current‚17039360 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.freed‚0 P@DeviceStatsMonitor.on_validation_batch_end/active_bytes.all.peak‚ 13955591168 TFDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.freed‚ 225003594 [MDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.current‚ 673144760 I@DeviceStatsMonitor.on_validation_batch_end/segment.all.allocated‚1171 G=DeviceStatsMonitor.on_validation_batch_end/active.all.current‚20521 WJDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_validation_batch_end/allocation.all.allocated‚ 498223061 93DeviceStatsMonitor.on_validation_batch_end/num_ooms‚0 trainer/global_step‚10880 ZMDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.current‚18002944 RLDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.current‚2 TLDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.current‚209 RIDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.peak‚1187 WGDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.allocated‚ 14220787712 _runtime‚12725.912323044 YEDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.allocated‚908624800262144 JDDeviceStatsMonitor.on_validation_batch_end/active.large_pool.current‚2 QCDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.freed‚ 205381152 YJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.peak‚ 1175007241 `LDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.allocated‚896240784820736 [MDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.current‚ 675116544 M?DeviceStatsMonitor.on_validation_batch_end/allocation.all.freed‚ 498202540 GADeviceStatsMonitor.on_validation_batch_end/oversize_segments.peak‚0 UEDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.current‚ 14220787712 ICDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.freed‚0 TFDeviceStatsMonitor.on_validation_batch_end/active.large_pool.allocated‚ 225003596 UGDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.allocated‚ 205381363 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.freed‚65469707 XJDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.allocated‚ 273219465 MGDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.current‚0 JDDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.peak‚0 OEDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.peak‚31686 ICDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.freed‚0 TFDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.current‚ 693119488 _KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.freed‚896240766817792 bODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.allocated‚12384015441408 ^NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 YIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 XIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 PIDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.peak‚35 XDDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.freed‚908624107142656 YJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.peak‚ 1177047040 _RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.current‚23940096 KEDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.freed‚0 KADeviceStatsMonitor.on_validation_batch_end/allocation.all.current‚20521 ICDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.freed‚0 MEDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.current‚607 _step‚893 B&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 particle/accuracy_1_of_2‚1 :"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.44999998807907104 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9340277910232544 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 :"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.49427586793899536 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.13490022718906403 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.29395073652267456 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.28156328201293945 :"Purity/2FRt*SRqqt2FBt/event_purity‚0.03488372266292572 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 7 Purity/0FRt3SRqqt1FBt/FBt_purity‚0.8888888955116272 &particle/f_score‚0.437934011220932 *"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.5 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6463916897773743 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.4833333194255829 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 :"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.48773154616355896 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 particle/accuracy_3_of_3‚1 ;"Purity/0FRt1SRqqt2FBt/event_purity‚0.035919543355703354 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.5583332777023315 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 jet/accuracy_6_of_6‚0 ;"Purity/*FRt3SRqqt0FBt/event_purity‚0.008064515888690948 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.2773251235485077 :"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.42897897958755493 ("Purity/4FRt*SRqqt0FBt/SRqqt_purity‚1 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9871900677680969 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.4722222089767456 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 * Purity/*FRt*SRqqt4FBt/FBt_purity‚0.875 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.1666666716337204 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.5115677714347839 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 8 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.47093021869659424 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.4477556347846985 ;"Purity/1FRt1SRqqt1FBt/event_purity‚0.036805007606744766 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 jet/accuracy_5_of_7‚0 particle/accuracy_2_of_7‚1 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 9"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.4214073121547699 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9974767565727234 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9922967553138733 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.4309813678264618 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.017506100237369537 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.4005811810493469 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.1093914806842804 )jet/accuracy_1_of_4‚0.924567461013794 particle/accuracy_3_of_5‚1 *"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.5 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8159980177879333 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.030574452131986618 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8888888955116272 epoch‚8 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.3429912328720093 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.27281811833381653 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.49181386828422546 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.21861648559570312 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.18879307806491852 8 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.38235294818878174 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.4943356215953827 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.22837255895137787 9"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.4224596321582794 jet/accuracy_7_of_7‚0 ,"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.625 & Purity/2FRt2SRqqt2FBt/FRt_purity‚0 trainer/global_step‚10880 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9983713626861572 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9906989336013794 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.1944444477558136 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5660274624824524 :"Purity/0FRt2SRqqt1FBt/event_purity‚0.01886792480945587 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 9"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.4092593193054199 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 particle/accuracy_1_of_4‚1 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.4963136911392212 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.4722222089767456 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.3333333432674408 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 & Purity/*FRt3SRqqt2FBt/FRt_purity‚0 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9243499040603638 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.3437500596046448 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.3333333432674408 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 8"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.443687379360199 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.22312557697296143 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.25184008479118347 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ;"Purity/0FRt3SRqqt*FBt/event_purity‚0.026143791154026985 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.3928571343421936 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.27223947644233704 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9922183752059937 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.4652777910232544 & Purity/3FRt4SRqqt*FBt/FRt_purity‚0 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.6111111044883728 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.5944984555244446 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 7 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.2824074327945709 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5288735032081604 :"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.45562615990638733 +particle/specificity‚0.9504297375679016 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 particle/accuracy_2_of_6‚1 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 8 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.39998260140419006 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 7 Purity/1FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 +jet/accuracy_4_of_5‚0.01711469516158104 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.5614035129547119 jet/accuracy_1_of_6‚1 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 :"Purity/0FRt0SRqqt2FBt/event_purity‚0.33666330575942993 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.2523540258407593 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 *jet/accuracy_1_of_2‚0.7414181232452393 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9877815842628479 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 particle/accuracy_4_of_6‚1 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.4091647267341614 particle/accuracy_2_of_5‚1 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.20000000298023224 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9913264513015747 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 +"Purity/1FRt*SRqqt3FBt/SRqqt_purity‚0.75 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9938936829566956 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.2700437605381012 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 particle/accuracy_5_of_6‚1 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.3888889253139496 6 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.336866557598114 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.5098684430122375 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 4validation_average_jet_accuracy‚0.41836017370224 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 6 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.988541305065155 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.7142857313156128 particle/accuracy_4_of_7‚1 ("Purity/4FRt*SRqqt*FBt/SRqqt_purity‚1 :"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.46656569838523865 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.4736842215061188 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.1626984030008316 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9759984612464905 particle/accuracy_3_of_7‚1 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 _runtime‚12741.538009395 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.27743053436279297 8 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.27409908175468445 ,jet/accuracy_4_of_4‚0.004207609221339226 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 :"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.43738415837287903 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9923410415649414 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 :"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.49362871050834656 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9996549487113953 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.22590185701847076 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.018316958099603653 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9338625073432922 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.45861583948135376 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.5333333611488342 ) Purity/4FRt1SRqqt0FBt/FRt_purity‚0.25 :"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.41907793283462524 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9994331002235413 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.4729121923446655 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.03606931120157242 7 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.2100907266139984 * Purity/*FRt2SRqqt2FBt/FRt_purity‚0.125 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 particle/accuracy_1_of_7‚1 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 8 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.13333334028720856 7 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.8571428656578064 ) Purity/*FRt1SRqqt3FBt/FRt_purity‚0.75 -"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.5625 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9916117787361145 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9979475140571594 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.6666666865348816 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.1666666716337204 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.4227588474750519 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 8"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.460513174533844 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.23909702897071838 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.9093567728996277 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 +jet/accuracy_3_of_6‚0.11538461595773697 :"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.49666666984558105 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2962508201599121 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.28292182087898254 jet/accuracy_4_of_6‚0 _step‚894 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9961150288581848 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 jet/accuracy_5_of_6‚0 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9351851940155029 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.2257489711046219 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 particle/accuracy_1_of_5‚1 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.2581644356250763 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0æ¯!µìM.4527297914028168 9"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.5015432238578796 & Purity/1FRt0SRqqt4FBt/FRt_purity‚0 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.5245149731636047 9"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.4279460906982422 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.30002978444099426 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 8 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.39772728085517883 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 particle/accuracy_4_of_5‚1 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.3958333432674408 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.4285714328289032 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9898269176483154 particle/accuracy_5_of_5‚1 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.09210526198148727 , Purity/1FRt3SRqqt1FBt/FRt_purity‚0.28125 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 jet/accuracy_5_of_5‚0 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 8 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.25462964177131653 7 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.2561728358268738 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9986680150032043 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 * Purity/4FRt*SRqqt*FBt/FRt_purity‚0.125 *jet/accuracy_1_of_1‚0.4488156735897064 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.4218846261501312 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 <"Purity/1FRt2SRqqt0FBt/event_purity‚0.0024420025292783976 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 7 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.2750000059604645 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.18648989498615265 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9969576597213745 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2944791615009308 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 :"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.44999998807907104 9"Purity/*FRt2SRqqt*FBt/event_purity‚0.0265511367470026 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9940325617790222 7 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.9166666865348816 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.13475176692008972 :"Purity/0FRt*SRqqt*FBt/event_purity‚0.43201830983161926 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2093253880739212 jet/accuracy_1_of_7‚1 particle/accuracy_7_of_7‚1 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 particle/accuracy_6_of_6‚1 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 9"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.3333333432674408 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.6101190447807312 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 *jet/accuracy_3_of_5‚0.1292242705821991 :"Purity/0FRt*SRqqt0FBt/event_purity‚0.32058414816856384 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 particle/accuracy_3_of_6‚1 7 Purity/3FRt1SRqqt2FBt/FRt_purity‚0.1666666716337204 & Purity/3FRt0SRqqt2FBt/FRt_purity‚0 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.2522985637187958 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 *"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.5 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.2422630488872528 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.21799729764461517 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 particle/accuracy_4_of_4‚1 & Purity/*FRt*SRqqt4FBt/FRt_purity‚0 9"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.6000000238418579 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.4041047692298889 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9977324604988098 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 particle/accuracy_2_of_4‚1 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 :"Purity/*FRt*SRqqt1FBt/event_purity‚0.40659651160240173 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 particle/accuracy_1_of_1‚1 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 ("Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.07510586827993393 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.06637628376483917 :"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.45952460169792175 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 jet/accuracy_2_of_7‚0.5 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.1911889761686325 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9987789988517761 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 particle/accuracy_1_of_3‚1 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 8 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.13333334028720856 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.009542706422507763 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.4563834071159363 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_2_of_2‚1 +"Purity/1FRt1SRqqt3FBt/SRqqt_purity‚0.75 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 8 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.44963520765304565 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.5244253277778625 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.2618549168109894 <"Purity/1FRt2SRqqt*FBt/event_purity‚0.0021919880528002977 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 +particle/sensitivity‚0.3556158244609833 7 Purity/*FRt3SRqqt*FBt/FBt_purity‚0.9642857313156128 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.25809943675994873 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 & Purity/3FRt4SRqqt0FBt/FRt_purity‚0 +"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.25 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.4110149145126343 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.4027777910232544 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.10074356943368912 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.2257489711046219 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.4705459773540497 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.3311084806919098 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 *jet/accuracy_2_of_4‚0.3527919352054596 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9942528009414673 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 :"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.40762531757354736 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.5714285969734192 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 :"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.41851815581321716 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 *jet/accuracy_1_of_3‚0.8715117573738098 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 *"Purity/0FRt2SRqqt3FBt/SRqqt_purity‚0.5 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.18187135457992554 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 *"Purity/*FRt2SRqqt3FBt/SRqqt_purity‚0.5 7 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.2252924144268036 particle/accuracy_1_of_6‚1 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.013557745143771172 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.5015725493431091 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 *jet/accuracy_2_of_5‚0.4730031192302704 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.20000000298023224 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 * Purity/1FRt0SRqqt3FBt/FRt_purity‚0.375 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.013020375743508339 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 particle/accuracy_5_of_7‚1 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚1 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9907812476158142 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.3331485986709595 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9774305820465088 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.29918912053108215 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.5307602286338806 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.3861788511276245 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 :"Purity/*FRt1SRqqt2FBt/event_purity‚0.01912442408502102 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 particle/accuracy_2_of_3‚1 6 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.095238097012043 7 Purity/1FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.17852912843227386 6 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.949999988079071 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5211085677146912 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.4327617585659027 :"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.43209874629974365 ;"Purity/0FRt3SRqqt0FBt/event_purity‚0.026143791154026985 & Purity/0FRt2SRqqt2FBt/FBt_purity‚1 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 8"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.696969747543335 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 *"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.5 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.16620036959648132 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 ,jet/accuracy_3_of_3‚0.020242666825652122 +"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.25 7 Purity/0FRt3SRqqt*FBt/FBt_purity‚0.8888888955116272 8"Purity/*FRt*SRqqt*FBt/event_purity‚0.235612154006958 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.2674388587474823 9"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.4153256118297577 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 9"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.4153595268726349 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 jet/accuracy_3_of_7‚0 jet/accuracy_4_of_7‚0 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.2365124374628067 + Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9375 :"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.44323089718818665 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 *jet/accuracy_1_of_5‚0.9542178511619568 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.4583333134651184 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.21206897497177124 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 particle/accuracy_6_of_7‚1 & Purity/1FRt*SRqqt4FBt/FRt_purity‚0 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 ;"Purity/*FRt3SRqqt*FBt/event_purity‚0.008064515888690948 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 jet/accuracy_6_of_7‚0 :"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.40657204389572144 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.3947804868221283 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.21911375224590302 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9941520690917969 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.006728632375597954 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 8"Purity/*FRt1SRqqt0FBt/event_purity‚0.208349347114563 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9900730848312378 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.3947804868221283 +jet/accuracy_2_of_2‚0.10519388318061829 :"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.48666664958000183 & Purity/1FRt3SRqqt2FBt/FRt_purity‚0 ) Purity/4FRt1SRqqt*FBt/FRt_purity‚0.25 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.1861431747674942 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5370014905929565 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.06577380746603012 *jet/accuracy_2_of_6‚0.4294871687889099 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 ,jet/accuracy_3_of_4‚0.061950914561748505 'particle/accuracy‚0.857556164264679 $ _timestamp‚1.788523116065291e+09 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.4228139817714691 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.22188284993171692 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9878469109535217 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9908074736595154 /"Purity/0FRt0SRqqt3FBt/event_purity‚0.109375 * Purity/0FRt0SRqqt4FBt/FBt_purity‚0.875 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 8 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.29933857917785645 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.10785634815692902 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.34732142090797424 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 ,"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.625 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚1 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.24247758090496063 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 7 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.2782828211784363 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 ."Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.40625 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.2336169332265854 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 7 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.2576628625392914 7 Purity/*FRt3SRqqt1FBt/FBt_purity‚0.9629629850387573 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 & Purity/*FRt0SRqqt4FBt/FRt_purity‚0 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 :"Purity/2FRt0SRqqt2FBt/event_purity‚0.05263157933950424 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.24071577191352844 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.09702705591917038 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9923497438430786 ."Purity/1FRt0SRqqt3FBt/event_purity‚0.21875 6 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.983730137348175 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 *"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.5 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.1666666716337204 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 +jet/accuracy_2_of_3‚0.24663276970386505 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.3989236056804657 ;"Purity/*FRt2SRqqt0FBt/event_purity‚0.030730580911040306 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.3630208373069763 particle/accuracy_3_of_4‚1 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6812902688980103 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 ) Purity/1FRt1SRqqt3FBt/FRt_purity‚0.75 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 7 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.2348485141992569 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.24222218990325928 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 *loss/val_total_loss‚38.037992170708655 * Purity/4FRt*SRqqt0FBt/FRt_purity‚0.125õ$±2“,ì¦ 7 Purity/0FRt3SRqqt1FBt/FBt_purity‚0.8888888955116272 &particle/f_score‚0.437934011220932 *"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.5 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6463916897773743 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.4833333194255829 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 :"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.48773154616355896 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 particle/accuracy_3_of_3‚1 ;"Purity/0FRt1SRqqt2FBt/event_purity‚0.035919543355703354 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.5583332777023315 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 jet/accuracy_6_of_6‚0 ;"Purity/*FRt3SRqqt0FBt/event_purity‚0.008064515888690948 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.2773251235485077 :"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.42897897958755493 ("Purity/4FRt*SRqqt0FBt/SRqqt_purity‚1 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9871900677680969 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.4722222089767456 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 * Purity/*FRt*SRqqt4FBt/FBt_purity‚0.875 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.1666666716337204 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.5115677714347839 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 8 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.47093021869659424 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.4477556347846985 ;"Purity/1FRt1SRqqt1FBt/event_purity‚0.036805007606744766 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 jet/accuracy_5_of_7‚0 particle/accuracy_2_of_7‚1 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 9"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.4214073121547699 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9974767565727234 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9922967553138733 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.4309813678264618 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.017506100237369537 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.4005811810493469 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.1093914806842804 )jet/accuracy_1_of_4‚0.924567461013794 particle/accuracy_3_of_5‚1 *"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.5 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8159980177879333 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.030574452131986618 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8888888955116272 epoch‚8 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.3429912328720093 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.27281811833381653 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.49181386828422546 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.21861648559570312 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.18879307806491852 8 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.38235294818878174 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.4943356215953827 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.22837255895137787 9"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.4224596321582794 jet/accuracy_7_of_7‚0 ,"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.625 & Purity/2FRt2SRqqt2FBt/FRt_purity‚0 trainer/global_step‚10880 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9983713626861572 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9906989336013794 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.1944444477558136 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5660274624824524 :"Purity/0FRt2SRqqt1FBt/event_purity‚0.01886792480945587 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 9"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.4092593193054199 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 particle/accuracy_1_of_4‚1 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.4963136911392212 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.4722222089767456 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.3333333432674408 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 & Purity/*FRt3SRqqt2FBt/FRt_purity‚0 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9243499040603638 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.3437500596046448 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.3333333432674408 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 8"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.443687379360199 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.22312557697296143 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.25184008479118347 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ;"Purity/0FRt3SRqqt*FBt/event_purity‚0.026143791154026985 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.3928571343421936 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.27223947644233704 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9922183752059937 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.4652777910232544 & Purity/3FRt4SRqqt*FBt/FRt_purity‚0 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.6111111044883728 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.5944984555244446 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 7 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.2824074327945709 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5288735032081604 :"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.45562615990638733 +particle/specificity‚0.9504297375679016 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 particle/accuracy_2_of_6‚1 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 8 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.39998260140419006 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 7 Purity/1FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 +jet/accuracy_4_of_5‚0.01711469516158104 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.5614035129547119 jet/accuracy_1_of_6‚1 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 :"Purity/0FRt0SRqqt2FBt/event_purity‚0.33666330575942993 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.2523540258407593 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 *jet/accuracy_1_of_2‚0.7414181232452393 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9877815842628479 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 particle/accuracy_4_of_6‚1 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.4091647267341614 particle/accuracy_2_of_5‚1 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.20000000298023224 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9913264513015747 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 +"Purity/1FRt*SRqqt3FBt/SRqqt_purity‚0.75 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9938936829566956 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.2700437605381012 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 particle/accuracy_5_of_6‚1 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.3888889253139496 6 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.336866557598114 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.5098684430122375 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 4validation_average_jet_accuracy‚0.41836017370224 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 6 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.988541305065155 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.7142857313156128 particle/accuracy_4_of_7‚1 ("Purity/4FRt*SRqqt*FBt/SRqqt_purity‚1 :"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.46656569838523865 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.4736842215061188 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.1626984030008316 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9759984612464905 particle/accuracy_3_of_7‚1 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 _runtime‚12741.538009395 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.27743053436279297 8 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.27409908175468445 ,jet/accuracy_4_of_4‚0.004207609221339226 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 :"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.43738415837287903 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9923410415649414 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 :"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.49362871050834656 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9996549487113953 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.22590185701847076 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.018316958099603653 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9338625073432922 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.45861583948135376 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.5333333611488342 ) Purity/4FRt1SRqqt0FBt/FRt_purity‚0.25 :"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.41907793283462524 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚ŒñÊPma0 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9994331002235413 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.4729121923446655 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.03606931120157242 7 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.2100907266139984 * Purity/*FRt2SRqqt2FBt/FRt_purity‚0.125 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 particle/accuracy_1_of_7‚1 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 8 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.13333334028720856 7 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.8571428656578064 ) Purity/*FRt1SRqqt3FBt/FRt_purity‚0.75 -"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.5625 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9916117787361145 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9979475140571594 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.6666666865348816 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.1666666716337204 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.4227588474750519 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 8"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.460513174533844 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.23909702897071838 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.9093567728996277 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 +jet/accuracy_3_of_6‚0.11538461595773697 :"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.49666666984558105 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2962508201599121 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.28292182087898254 jet/accuracy_4_of_6‚0 _step‚894 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9961150288581848 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 jet/accuracy_5_of_6‚0 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9351851940155029 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.2257489711046219 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 particle/accuracy_1_of_5‚1 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.2581644356250763 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.4527297914028168 9"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.5015432238578796 & Purity/1FRt0SRqqt4FBt/FRt_purity‚0 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.5245149731636047 9"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.4279460906982422 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.30002978444099426 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 8 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.39772728085517883 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 particle/accuracy_4_of_5‚1 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.3958333432674408 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.4285714328289032 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9898269176483154 particle/accuracy_5_of_5‚1 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.09210526198148727 , Purity/1FRt3SRqqt1FBt/FRt_purity‚0.28125 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 jet/accuracy_5_of_5‚0 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 8 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.25462964177131653 7 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.2561728358268738 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9986680150032043 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 * Purity/4FRt*SRqqt*FBt/FRt_purity‚0.125 *jet/accuracy_1_of_1‚0.4488156735897064 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.4218846261501312 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 <"Purity/1FRt2SRqqt0FBt/event_purity‚0.0024420025292783976 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 7 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.2750000059604645 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.18648989498615265 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9969576597213745 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2944791615009308 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 :"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.44999998807907104 9"Purity/*FRt2SRqqt*FBt/event_purity‚0.0265511367470026 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9940325617790222 7 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.9166666865348816 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.13475176692008972 :"Purity/0FRt*SRqqt*FBt/event_purity‚0.43201830983161926 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2093253880739212 jet/accuracy_1_of_7‚1 particle/accuracy_7_of_7‚1 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 particle/accuracy_6_of_6‚1 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 9"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.3333333432674408 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.6101190447807312 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 *jet/accuracy_3_of_5‚0.1292242705821991 :"Purity/0FRt*SRqqt0FBt/event_purity‚0.32058414816856384 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 particle/accuracy_3_of_6‚1 7 Purity/3FRt1SRqqt2FBt/FRt_purity‚0.1666666716337204 & Purity/3FRt0SRqqt2FBt/FRt_purity‚0 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.2522985637187958 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 *"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.5 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.2422630488872528 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.21799729764461517 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 particle/accuracy_4_of_4‚1 & Purity/*FRt*SRqqt4FBt/FRt_purity‚0 9"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.6000000238418579 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.4041047692298889 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9977324604988098 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 particle/accuracy_2_of_4‚1 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 :"Purity/*FRt*SRqqt1FBt/event_purity‚0.40659651160240173 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 particle/accuracy_1_of_1‚1 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 ("Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.07510586827993393 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.06637628376483917 :"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.45952460169792175 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 jet/accuracy_2_of_7‚0.5 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.1911889761686325 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9987789988517761 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 particle/accuracy_1_of_3‚1 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 8 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.13333334028720856 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.009542706422507763 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.4563834071159363 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_2_of_2‚1 +"Purity/1FRt1SRqqt3FBt/SRqqt_purity‚0.75 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 8 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.44963520765304565 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.5244253277778625 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.2618549168109894 <"Purity/1FRt2SRqqt*FBt/event_purity‚0.0021919880528002977 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 +particle/sensitivity‚0.3556158244609833 7 Purity/*FRt3SRqqt*FBt/FBt_purity‚0.9642857313156128 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.25809943675994873 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 & Purity/3FRt4SRqqt0FBt/FRt_purity‚0 +"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.25 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.4110149145126343 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.4027777910232544 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.10074356943368912 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.2257489711046219 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.4705459773540497 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.3311084806919098 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 *jet/accuracy_2_of_4‚0.3527919352054596 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9942528009414673 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 :"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.40762531757354736 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.5714285969734192 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 :"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.41851815581321716 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 *jet/accuracy_1_of_3‚0.8715117573738098 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 *"Purity/0FRt2SRqqt3FBt/SRqqt_purity‚0.5 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.18187135457992554 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 *"Purity/*FRt2SRqqt3FBt/SRqqt_purity‚0.5 7 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.2252924144268036 particle/accuracy_1_of_6‚1 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.013557745143771172 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.5015725493431091 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 *jet/accuracy_2_of_5‚0.4730031192302704 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.20000000298023224 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 * Purity/1FRt0SRqqt3FBt/FRt_purity‚0.375 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.013020375743508339 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 particle/accuracy_5_of_7‚1 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚1 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9907812476158142 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.3331485986709595 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9774305820465088 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.29918912053108215 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.5307602286338806 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.3861788511276245 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 :"Purity/*FRt1SRqqt2FBt/event_purity‚0.01912442408502102 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 particle/accuracy_2_of_3‚1 6 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.095238097012043 7 Purity/1FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.17852912843227386 6 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.949999988079071 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5211085677146912 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.4327617585659027 :"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.43209874629974365 ;"Purity/0FRt3SRqqt0FBt/event_purity‚0.026143791154026985 & Purity/0FRt2SRqqt2FBt/FBt_purity‚1 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 8"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.696969747543335 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 *"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.5 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.16620036959648132 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 ,jet/accuracy_3_of_3‚0.020242666825652122 +"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.25 7 Purity/0FRt3SRqqt*FBt/FBt_purity‚0.8888888955116272 8"Purity/*FRt*SRqqt*FBt/event_purity‚0.235612154006958 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.2674388587474823 9"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.4153256118297577 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 9"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.4153595268726349 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 jet/accuracy_3_of_7‚0 jet/accuracy_4_of_7‚0 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.2365124374628067 + Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9375 :"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.44323089718818665 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 *jet/accuracy_1_of_5‚0.9542178511619568 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.4583333134651184 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.21206897497177124 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 particle/accuracy_6_of_7‚1 & Purity/1FRt*SRqqt4FBt/FRt_purity‚0 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 ;"Purity/*FRt3SRqqt*FBt/event_purity‚0.008064515888690948 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 jet/accuracy_6_of_7‚0 :"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.40657204389572144 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.3947804868221283 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.21911375224590302 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9941520690917969 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.006728632375597954 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 8"Purity/*FRt1SRqqt0FBt/event_purity‚0.208349347114563 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9900730848312378 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.3947804868221283 +jet/accuracy_2_of_2‚0.10519388318061829 :"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.48666664958000183 & Purity/1FRt3SRqqt2FBt/FRt_purity‚0 ) Purity/4FRt1SRqqt*FBt/FRt_purity‚0.25 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.1861431747674942 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5370014905929565 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.06577380746603012 *jet/accuracy_2_of_6‚0.4294871687889099 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 ,jet/accuracy_3_of_4‚0.061950914561748505 'particle/accuracy‚0.857556164264679 $ _timestamp‚1.788523116065291e+09 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.4228139817714691 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.22188284993171692 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9878469109535217 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9908074736595154 /"Purity/0FRt0SRqqt3FBt/event_purity‚0.109375 * Purity/0FRt0SRqqt4FBt/FBt_purity‚0.875 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 8 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.29933857917785645 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.10785634815692902 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.34732142090797424 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 ,"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.625 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚1 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.24247758090496063 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 7 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.2782828211784363 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 ."Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.40625 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.2336169332265854 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 7 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.2576628625392914 7 Purity/*FRt3SRqqt1FBt/FBt_purity‚0.9629629850387573 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 & Purity/*FRt0SRqqt4FBt/FRt_purity‚0 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 :"Purity/2FRt0SRqqt2FBt/event_purity‚0.05263157933950424 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.24071577191352844 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.09702705591917038 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9923497438430786 ."Purity/1FRt0SRqqt3FBt/event_purity‚0.21875 6 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.983730137348175 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 *"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.5 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.1666666716337204 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 +jet/accuracy_2_of_3‚0.24663276970386505 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.3989236056804657 ;"Purity/*FRt2SRqqt0FBt/event_purity‚0.030730580911040306 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.3630208373069763 particle/accuracy_3_of_4‚1 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6812902688980103 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 ) Purity/1FRt1SRqqt3FBt/FRt_purity‚0.75 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 7 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.2348485141992569 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.24222218990325928 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 *loss/val_total_loss‚38.037992170708655 * Purity/4FRt*SRqqt0FBt/FRt_purity‚0.125 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9994331002235413 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.4551856815814972 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.4491255283355713 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.20690308511257172 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 9"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.4444444179534912 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.06571291387081146 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.2816549241542816 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 particle/accuracy_1_of_2‚1 :"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.44999998807907104 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9340277910232544 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 :"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.49427586793899536 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.13490022718906403 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.29395073652267456 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.28156328201293945 :"Purity/2FRt*SRqqt2FBt/event_purity‚0.03488372266292572 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658þÖuÙë’”,‚(:‡ ôäêÔ¿þ΀ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚35.0 gpu.0.powerWatts‚113.798' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.4495 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0næÑRf•,‚(:Û ôäêÔÖŸ­ƒ disk.nvme0n1.in‚ 619.63671875 disk.sda.in‚ 10.7578125$ memory_percent‚92.09058284759521 disk.sdb2.in‚ 7.0546875$ disk./.usageGB‚2713.625442504883 disk.sdb.in‚ 7.0546875 disk.sdb.out‚1458.7138671875 disk.sdb1.in‚0 disk.sda1.in‚0! disk.sda2.out‚1458.7138671875 disk.rbd0.in‚0 disk.nvme1n1.out‚ 67131.90625 disk.md0.in‚17.8125 disk.rbd2.out‚ 3.529296875) proc.memory.availableMB‚ 1295.87890625 disk.sda1.out‚0 disk.dm-0.out‚ 143361.28125# proc.memory.rssMB‚ 2992.45703125 proc.cpu.threads‚17 network.recv‚ 4.095791e+06( disk./.usagePercent‚75.8741031129175 disk.rbd0.out‚ 3.591796875 disk.sdb1.out‚0 disk.rbd2.in‚0 disk.dm-0.in‚ 1118.4375) proc.memory.percent‚18.26450824737549 cpu‚25.220024784465444 disk.md0p1.in‚17.8125 disk.md0p1.out‚ 1406.3515625 disk.nvme0n1.out‚ 76230.125 disk.sda.out‚1458.7138671875 disk.md0.out‚ 1406.3515625 network.sent‚ 2.8634277e+07 disk.sda2.in‚ 10.7578125 disk.nvme1n1.in‚ 498.80078125! disk.sdb2.out‚1458.7138671875©4–,/ _wandbruntime‚12762  _runtime‚12762L@¹Ë—,Å % _timestamp‚1.7885231372235825e+09 _runtime‚12762.694041357 _step‚895 ' lr-AdamW/pg1‚2.5106009465825808e-05 ' lr-AdamW/pg2‚2.5106009465825808e-05 trainer/global_step‚108993µ26И,Ê % _timestamp‚1.7885231372235825e+09 _runtime‚12762.694041357 _step‚895 ' lr-AdamW/pg1‚2.5106009465825808e-05 ' lr-AdamW/pg2‚2.5106009465825808e-05 trainer/global_step‚10899ÿhúnã4™,/ _wandbruntime‚12762  _runtime‚12762 fŽº`š,†P LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 225560346 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26088 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955591168 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚580 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12409508260864 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12410407668736 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚910780046983680 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚898370556725760 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917410816 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚910780046983680 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 244411392 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1187 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12962117383168 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214665 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 205895035 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1212 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 499341056 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1018001552824320 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26090 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 401960960 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚890587481252692 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12363254241764 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚12762.694954241 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚898370556725760 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411420 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007241 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410816 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 273754620 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1005039655912448 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 220471296 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177047040 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 225560348 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 205895617 fQDeviceStatsMonitor.on_tr9iö¬ain_batch_start/inactive_split_bytes.large_pool.allocated‚1005039679852544 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 % _timestamp‚1.7885231372244468e+09 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 140256935 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26088 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚65638682 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚878224227010928 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 225560348 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407872 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚898370538722816 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955591168 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1018001797235712 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚890588395703472 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450780 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12364151653184 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚910780964394496 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚910780964394496 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407872 trainer/global_step‚10899 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 225560346 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411420 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007241 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410816 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 273754620 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1005039655912448 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 220471296 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177047040 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 225560348 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 205895617 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1005039679852544 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 % _timestamp‚1.7885231372244468e+09 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 140256935 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26088 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚65638682 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚878224227010928 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 225560348 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407872 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚898370538722816 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955591168 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1018001797235712 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚890588395703472 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450780 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12364151653184 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚910780964394496 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚910780964394496 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407872 trainer/global_step‚10899 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 225560346 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚910780046983680 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚898370556725760 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917410816 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚910780046983680 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 244411392 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1187 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12962117383168 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214665 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 205895035 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1212 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 499341056 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1018001552824320 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26090 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 401960960 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚890587481252692 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12363254241764€=x”_4œ,/ _wandbruntime‚12763  _runtime‚12763T<ÜR',N >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12365260314710 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 273805325 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚898450187021312 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 225580246 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚890667484459126 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1005128975133184 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12963047783424 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚878302207105056 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12411520485376 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚898450187021312 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 225580246 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 254883840 _step‚897 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1005128999073280 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12364362903082 _runtime‚12763.699512512 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26114 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955591168 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177047040 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚602 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 205912899 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚10899 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 140269311 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚898450169018368 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 499385571 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 499359455 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 273805325 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1018092277800448 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚878302224144416 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177047040 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚910861707506688 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 225580244 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚898450169018368 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899421184 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12410621064192 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0e†æמ,”N HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 499359455 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12411520485376 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 140269309 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450988 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚890666570008138 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917424128 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411628 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 $ _timestamp‚1.788523138227115e+09 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899421184 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12410621064192 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12365260314710 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 273805325 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚898450187021312 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 225580246 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚890667484459126 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1005128975133184 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12963047783424 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚878302207105056 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12411520485376 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚898450187021312 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 225580246 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 254883840 _step‚897 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1005128999073280 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12364362903082 _runtime‚12763.699512512 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26114 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955591168 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177047040 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚602 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 205912899 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚10899 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 140269311 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚898450169018368 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 499385571 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 499359455 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 273805325 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1018092277800448 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚878302224144416 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177047040 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚910861707506688 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 225580244 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚898450169018368 FDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 275013610 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚65919425 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 _runtime‚12820.645241673 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚914818133752320 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 140875735 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 254897152 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12418700898872 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12465162841088 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450780 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411420 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26088 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 206795158 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1187 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚914817216341504 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 501542768 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚894541932616200 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214665 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 226555248 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1022525244587008 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚65920047 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1022525499484160 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚882123248756688 _step‚900 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1009505640889344 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 % _timestamp‚1.7885231951748035e+09 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 274987522 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚902352071503360 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410816 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚622 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 230957056 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚902352053500416 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12465162841088 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177047040 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12419598310292 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 trainer/global_step‚10949 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 226555246 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26088 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407872 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 140875733 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955591168 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 274987522 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13019858594816 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚894542847066980 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 501568858 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚624 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 401960960 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚902352071503360 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007241 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13019627637760 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 206795782 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407872 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26090 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917410816 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚914818133752320 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚914817216341504 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 226555248 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚902352053500416 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 226555246 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12466062248960 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 275013610 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚65919425 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 _runtime‚12820.645241673 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚914818133752320 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 140875735 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 254897152 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12418700898872 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12465162841088 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450780 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411420 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26088 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 206795158 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1187 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚914817216341504 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 501542768 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚894541932616200 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214665 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 226555248 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1022525244587008 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚65920047 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1022525499484160 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚882123248756688 _step‚900 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1009505640889344 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 % _timestamp‚1.7885231951748035e+09 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 274987522 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚902352071503360 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410816 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚622 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 230957056 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚902352053500416 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12465162841088 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177047040 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12419598310292 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 trainer/global_step‚10949 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 226555246 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26088 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407872 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 140875733 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955591168 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 274987522 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13019858594816 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚894542847066980 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 501568858 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚624 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 401960960 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚902352071503360 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007241 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13019627637760 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 206795782 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407872 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26090 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917410816 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚914818133752320 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚914817216341504 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 226555248 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚902352053500416„8|B4°,/ _wandbruntime‚12821  _runtime‚12821+­£þï±,N OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 275038227 % _timestamp‚1.7885231962574928e+09 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 501587257 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 275038227 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31686 78â&%WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 140888111 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚65925563 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚914898876864512 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚902431701798912 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 252786688 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411628 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚914898876864512 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚902431683795968 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 trainer/global_step‚10949 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1022615717849600 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12420706971818 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 _runtime‚12821.72859048 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450988 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13021010526208 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917424128 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899421184 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 206813057 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚894621935822634 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 _step‚901 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 501613373 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚617 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚615 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 401960960 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12419809560190 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177047040 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 206813674 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 226575146 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 501587257 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚65924948 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚894621021371646 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 275012113 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 226575146 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26114 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚882201228850816 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007241 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12467175065600 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13020781679616 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450988 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13021010526208 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917424128 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899421184 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 206813057 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚894621935822634 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 _step‚901 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 501613373 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚617 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚615 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 401960960 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12419809560190 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177047040 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 206813674 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 226575146 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 501587257 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚65924948 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚894621021371646 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 275012113 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 226575146 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26114 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚882201228850816 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007241 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12467175065600 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13020781679616 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 252786688 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411628 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚914898876864512 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚902431683795968 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 trainer/global_step‚10949 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1022615717849600 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12420706971818 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171…Ì‚NÇ4³,/ _wandbruntime‚12821  _runtime‚12821Ç’£GY´,Ó 0loss/FRt3/assignment_loss‚2.2028413203027513 1loss/FBt1/assignment_loss‚0.36100082921459237 .loss/FRt1/detection_loss‚2.016029924484767 .loss/FRt4/assignment_loss‚3.10243479644551 1loss/SRqqt4/detection_loss‚2.1607619379505967 2loss/SRqqt3/assignment_loss‚1.4023724956547512 1loss/SRqqt1/detection_loss‚1.4437826592475176 /loss/FBt2/detection_loss‚0.5504981647925765 %loss/total_loss‚37.57919945772463 2loss/SRqqt1/assignment_loss‚1.0389455747242797 0loss/FBt2/assignment_loss‚0.1583279416832901 0loss/FBt3/assignment_loss‚0.2585736086739113 .loss/FRt4/detection_loss‚2.574172814102734 epoch‚9 _runtime‚12821.729527784 0loss/FRt1/assignment_loss‚2.2376260102963914 /loss/FRt2/assignment_loss‚1.375442665919923 2loss/SRqqt2/assignment_loss‚0.9523539803049061 /loss/FRt2/detection_loss‚1.4666214946069216 /loss/FBt3/detection_loss‚2.4955288648605345 $ _timestamp‚1.788523196259373e+09 .loss/FBt4/detection_loss‚2.683999706398357 1loss/SRqqt4/assignment_loss‚1.598303192041137 1loss/FBt4/assignment_loss‚0.34002583490258653 1loss/SRqqt2/detection_loss‚0.7897458516235929 0loss/SRqqt3/detection_loss‚1.640255035576867 .loss/FBt1/detection_loss‚2.656845417287615 _step‚902 /loss/FRt3/detection_loss‚2.0727093366285163 trainer/global_step‚10949[<‰y^µ,Ø 2loss/SRqqt3/assignment_loss‚1.4023724956547512 1loss/SRqqt1/detection_loss‚1.4437826592475176 /loss/FBt2/detection_loss‚0.5504981647925765 %loss/total_loss‚37.57919945772463 2loss/SRqqt1/assignment_loss‚1.0389455747242797 0loss/FBt2/assignment_loss‚0.1583279416832901 0loss/FBt3/assignment_loss‚0.2585736086739113 .loss/FRt4/detection_loss‚2.574172814102734 epoch‚9 _runtime‚12821.729527784 0loss/FRt1/assignment_loss‚2.2376260102963914 /loss/FRt2/assignment_loss‚1.375442665919923 2loss/SRqqt2/assignment_loss‚0.9523539803049061 /loss/FRt2/detection_loss‚1.4666214946069216 /loss/FBt3/detection_loss‚2.4955288648605345 $ _timestamp‚1.788523196259373e+09 .loss/FBt4/detection_loss‚2.683999706398357 1loss/SRqqt4/assignment_loss‚1.598303192041137 1loss/FBt4/assignment_loss‚0.34002583490258653 1loss/SRqqt2/detection_loss‚0.7897458516235929 0loss/SRqqt3/detection_loss‚1.640255035576867 .loss/FBt1/detection_loss‚2.656845417287615 _step‚902 /loss/FRt3/detection_loss‚2.0727093366285163 trainer/global_step‚10949 0loss/FRt3/assignment_loss‚2.2028413203027513 1loss/FBt1/assignment_loss‚0.36100082921459237 .loss/FRt1/detection_loss‚2.016029924484767 .loss/FRt4/assignment_loss‚3.10243479644551 1loss/SRqqt4/detection_loss‚2.1607619379505967†»]qÖi¶,‚(:Þ ¿åêÔžýÅ disk.sdb2.in‚ 7.0546875# disk.nvme0n1.out‚77615.96484375 disk.rbd2.in‚0 disk.rbd2.out‚ 3.529296875! disk.dm-0.out‚144749.33203125 disk.md0p1.out‚ 1414.5625 disk.sdb.in‚ 7.0546875 disk.sdb1.in‚0 disk.sdb1.out‚0! disk.sdb2.out‚1467.2626953125 disk.sda1.in‚0 disk.sda.out‚1467.2626953125" disk.nvme1n1.out‚ 67133.3671875 disk.sda1.out‚0 disk.sda.in‚ 10.7578125! disk.sda2.out‚1467.2626953125$ memory_percent‚92.09737777709961' proc.memory.availableMB‚ 1294.765625) proc.memory.percent‚18.26450824737549 disk.sda2.in‚ 10.7578125 disk.md0.in‚17.8125 disk.nvme0n1.in‚ 619.63671875% disk./.usageGB‚2713.8736305236816 disk.dm-0.in‚ 1118.4375 disk.md0p1.in‚17.8125 disk.sdb.out‚1467.2626953125 disk.rbd0.in‚0 disk.rbd0.out‚ 3.591796875 disk.nvme1n1.in‚ 498.80078125 disk.md0.out‚ 1414.5625 network.sent‚ 2.8894997e+07) disk./.usagePercent‚75.88104255379787# proc.memory.rssMB‚ 2992.45703125 cpu‚25.21888412695275 proc.cpu.threads‚17 network.recv‚ 4.119685e+06HòÌ{“·,‚(:ˆ ¿åêÔ¿ËÖ gpu.0.gpu‚43.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚36.0 gpu.0.powerWatts‚116.306' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚29.0765 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Çm¨l’¸,‚(:‡ ÎåêÔ¹”¼ÿ gpu.0.gpu‚3.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚37.0 gpu.0.powerWatts‚161.254' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚40.3135 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ŽéôLp¹,‚(:å ÎåêÔΓ¡€ disk.nvme1n1.in‚ 498.80078125 disk.sdb1.in‚0 disk.md0.out‚ 1416.06640625 disk.sda1.in‚0# proc.memory.rssMB‚ 2992.45703125) proc.memory.percent‚18.26450824737549 disk.sda.in‚ 10.7578125! disk.sda.out‚1468.82763671875 disk.rbd0.in‚0 disk.rbd2.in‚0 disk.sda1.out‚0! disk.sdb.out‚1468.82763671875$ disk./.usageGB‚2713.343475341797 disk.md0.in‚17.8125 disk.rbd2.out‚ 3.529296875$ memory_percent‚92.09742546081543" disk.sdb2.out‚1468.82763671875 disk.dm-0.in‚ 1118.4375 disk.sdb.in‚ 7.0546875 cpu‚25.218673092336548 network.recv‚ 4.126104e+06 disk.sda2.in‚ 10.7578125 disk.sdb1.out‚0 disk.nvme0n1.in‚ 619.63671875! disk.nvme0n1.out‚ 77868.453125( proc.memory.availableMB‚ 1294.7578125) disk./.usagePercent‚75.86621919302519! disk.nvme1n1.out‚ 67133.578125 disk.md0p1.in‚17.8125 disk.md0p1.out‚ 1416.06640625 proc.cpu.threads‚17 network.sent‚ 2.8990462e+07 disk.rbd0.out‚ 3.591796875 disk.sdb2.in‚ 7.0546875 disk.dm-0.out‚ 144978.15625" disk.sda2.out‚1468.82763671875"jˆ‚‘º,‚(:† ÝåêÔË–»ÿ gpu.0.gpu‚4.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚36.0 gpu.0.powerWatts‚114.408' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.602 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ƒNs»,‚(:è ÝåêÔ›©€ network.sent‚ 2.8995587e+07 disk.sdb1.out‚0 disk.rbd2.out‚ 3.529296875 disk.sdb2.in‚ 7.0546875 disk.md0p1.in‚17.8125 network.recv‚ 4.128881e+06 disk.rbd2.in‚0 disk.sda1.in‚0 disk.dm-0.out‚145021.2421875 disk.sdb.in‚ 7.0546875" disk.sda2.out‚1470.57177734375$ memory_percent‚92.09740161895752) proc.memory.availableMB‚ 1294.76171875# proc.memory.rssMB‚ 2992.45703125 disk.sda2.in‚ 10.7578125 disk.md0.in‚17.8125! disk.sdb.out‚1470.57177734375 disk.sda.in‚ 10.7578125 disk.rbd0.out‚ 3.591796875 disk.nvme1n1.in‚ 498.80078125) disk./.usagePercent‚75.86447539998554 disk.sdb1.in‚0" disk.sdb2.out‚1470.57177734375 disk.sda1.out‚0 disk.dm-0.in‚ 1118.4375) proc.memory.percent‚18.26450824737549# disk.nvme1n1.out‚67133.99609375 disk.nvme0n1.in‚ 619.63671875# disk.nvme0n1.out‚77887.24609375 proc.cpu.threads‚17$ disk./.usageGB‚2713.281108856201! disk.sda.out‚1470.57177734375 disk.rbd0.in‚0 disk.md0.out‚ 1417.734375 disk.md0p1.out‚ 1417.734375 cpu‚25.218456992857195Xú½Ñ“¼,‚(:ˆ ìåêÔÛ”®ÿ gpu.0.gpu‚37.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚168.822' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.2055 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0xâvMe½,‚(:Ú ìåêÔ—üêÿ disk.md0p1.in‚17.8125 disk.sda.in‚ 10.7578125 disk.sda.out‚ 1472.05078125 disk.rbd0.out‚ 3.591796875 disk.sdb1.out‚0 network.recv‚ 4.133312e+06 disk.nvme1n1.in‚ 498.80078125 disk.rbd2.out‚ 3.529296875# proc.memory.rssMB‚ 2992.45703125 cpu‚25.218243647669677) disk./.usagePercent‚75.86698245579305 disk.sdb2.in‚ 7.0546875 disk.dm-0.out‚ 145266.6875 disk.sdb.in‚ 7.0546875 disk.sdb1.in‚0 disk.rbd2.in‚0$ memory_percent‚92.09740161895752 disk.dm-0.in‚ 1118.4375 disk.md0p1.out‚ 1419.140625 disk.nvme0n1.in‚ 619.63671875 disk.md0.out‚ 1419.140625 network.sent‚ 2.9003156e+07# disk.nvme0n1.out‚78132.52734375 disk.md0.in‚17.8125 proc.cpu.threads‚17 disk.sda1.in‚0 disk.rbd0.in‚0 disk.sdb.out‚ 1472.05078125 disk.sda2.out‚ 1472.05078125) proc.memory.availableMB‚ 1294.76171875% disk./.usageGB‚2713.3707733154297 disk.sda2.in‚ 10.7578125# disk.nvme1n1.out‚67134.16015625) proc.memory.percent‚18.26450824737549 disk.sdb2.out‚ 1472.05078125 disk.sda1.out‚0…!4¾,/ _wandbruntime‚12878  _runtime‚12878SÜMßÉ¿,à _runtime‚12878.630178771 _step‚903 & lr-AdamW/pg1‚2.110053081275732e-05 & lr-AdamW/pg2‚2.110053081275732e-05 trainer/global_step‚10999 % _timestamp‚1.7885232531596937e+09ý`ùÎÀ,È _runtime‚12878.630178771 _step‚903 & lr-AdamW/pg1‚2.110053081275732e-05 & lr-AdamW/pg2‚2.110053081275732e-05 trainer/global_step‚10999 % _timestamp‚1.7885232531596937e+09‡ƒW„4Á,/ _wandbruntime‚12878  _runtime‚12878@‚õ^Â,…P VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚918855303110144 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 227550146 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 207696307 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13077510941696 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12520817421312 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 227550146 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 207696919 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚906333568278016 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 227550148 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚66202384 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 276220424 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚612 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1013971577986048 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955591168 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12520817421312 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚610 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26088 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12475044967400 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12474147555980 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚918854385699328 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1027049088927744 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 141494533 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚66201774 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 254897152 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 _runtime‚12878.63200988 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12521716829184 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚918854385699328 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1212 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917410816 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13077741898752 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚898496383979708 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚886022253463088 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 _step‚904 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 401960960 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1013971601926144 % _timestamp‚1.7885232531612997e+09 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 227550148 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686ªÞÃ(Ã,ŠP \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚906333568278016 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 227550148 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚66202384 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 276220424 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚612 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1013971577986048 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955591168 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12520817421312 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚610 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26088 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12475044967400 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12474147555980 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚918854385699328 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1027049088927744 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 141494533 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚66201774 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 254897152 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 _runtime‚12878.63200988 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12521716829184 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚918854385699328 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1212 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917410816 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13077741898752 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚898496383979708 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚886022253463088 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 _step‚904 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 401960960 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1013971601926144 % _timestamp‚1.7885232531612997e+09 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 227550148 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚918855303110144 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 227550146 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 207696307 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13077510941696 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12520817421312 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 227550146 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 207696919 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607ˆÖÕ~¨4Ä,/ _wandbruntime‚12879  _runtime‚12879ÌÍ•±'Å,N ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12522829645824 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 227570046 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 276245015 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1014060921146880 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚898575472735154 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 227570044 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 276245015 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚918936046222336 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚906413198573568 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12521930224640 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚607 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 141506911 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚906413216576512 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13078892596224 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13078667943936 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 % _timestamp‚1.7885232542366002e+09 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 276271129 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1187 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955591168 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 227570044 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 503841175 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚605 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12522829645824 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 207714208 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 141506909 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1027139565150720 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 _runtime‚12879.707782403 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 207714815 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚66207299 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚886100233557216 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚886100216517856 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12475256217298 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31686 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007241 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 503815059 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12521930224640 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917424128 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚906413216576512 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 227570046 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 503841175 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 248592384 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 trainer/global_step‚10999 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26114 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 401960960 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450988 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 11770470406ëJÖÆ,•N E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899421184 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 503841175 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚605 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12522829645824 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 207714208 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 141506909 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1027139565150720 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 _runtime‚12879.707782403 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 207714815 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚66207299 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚886100233557216 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚886100216517856 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12475256217298 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31686 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007241 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 503815059 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12521930224640 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917424128 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚906413216576512 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 227570046 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 503841175 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 248592384 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 trainer/global_step‚10999 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26114 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 401960960 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450988 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177047040 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12522829645824 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 227570046 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 276245015 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1014060921146880 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚898575472735154 P@DeviceStatsMonitor.on_train_batcÃn¸WEh_end/reserved_bytes.all.current‚ 14220787712 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 227570044 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 276245015 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚918936046222336 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚906413198573568 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12521930224640 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚607 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 141506911 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚906413216576512 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13078892596224 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13078667943936 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 % _timestamp‚1.7885232542366002e+09 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 276271129 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1187 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955591168 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 227570044 FDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12577371409408 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 505998372 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚922892472467968 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚910315083055616 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1031573098858496 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411420 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚66485500 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 228545048 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 228545046 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚889921241130128 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚889921258169488 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1212 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚922892472467968 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚620 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚902450835343216 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 277479414 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 506024462 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚618 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚902451749793996 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007241 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917410816 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 277479414 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 142113333 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1018437539022848 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚922891555057152 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407872 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26088 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚910315083055616 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177047040 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450780 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214665 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12529594213088 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26090 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚922891555057152 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12576472001536 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 208598215 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12576472001536 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 277453326 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26088 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 244411392 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1031572854447104 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 220471296 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410816 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚66484882 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 401960960 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407872 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0×B ÒB"×,‰P QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 228545048 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12577371409408 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 505998372 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚922892472467968 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚910315083055616 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1031573098858496 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411420 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚66485500 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 228545048 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 228545046 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚889921241130128 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚889921258169488 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1212 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚922892472467968 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚620 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚902450835343216 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 277479414 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 506024462 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚618 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚902451749793996 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007241 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917410816 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 277479414 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 142113333 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1018437539022848 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚922891555057152 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407872 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26088 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚910315083055616 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177047040 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450780 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214665 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12529594213088 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26090 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚922891555057152 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12576472001536 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 208598215 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12576472001536 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 277453326 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26088 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 244411392 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1031572854447104 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 220471296 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410816 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚66484882 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 401960960 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407872 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 208598835 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 228545046 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚910315101058560 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 277453326 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 E;DeviceStatsMonitor.on_train_batch_start/all`¨|ÈÍocation.all.peak‚31931 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 trainer/global_step‚11049 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚922972298156032 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚910394713351168 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 208616330 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12531600286034 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 228564944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚66491243 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1018526882183680 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177047040 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917424128 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13136476529664 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411628 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899421184 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12530702874406 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 _runtime‚12937.454600431 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 228564946 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 _step‚909 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12578484226048 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 506068977 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899421184 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 208616954 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 142125709 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1031663585462784 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1018526858243584 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚902529924098662 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 228564946 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26114 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚910394731354112 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13136703279104 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 506042861 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26114 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚902530838549650 $ _timestamp‚1.788523311983812e+09 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚910394713351168 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 506042861 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 277504031 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚889999238263616 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955591168 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450988 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0yêôx'Ú,”N K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 228564944 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚922972298156032 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚910394713351168 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 208616330 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12531600286034 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 228564944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚66491243 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1018526882183680 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177047040 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917424128 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13136476529664 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411628 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899421184 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12530702874406 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 _runtime‚12937.454600431 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 228564946 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 _step‚909 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12578484226048 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 506068977 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899421184 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 208616954 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 142125709 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1031663585462784 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1018526858243584 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚902529924098662 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 228564946 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26114 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚910394731354112 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13136703279104 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 506042861 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26114 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚902530838549650 $ _timestamp‚1.788523311983812e+09 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚910394713351168 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 506042861 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 277504031 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚889999238263616 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955591168 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450988 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚889999221224256 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 226749440 trainer/global_step‚11049 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 506068977 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚922973215580160 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31686 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 401960960 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚922973215580160 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 277477917 PÿW”à,‚(:‰ ÆæêÔÏž²ÿ gpu.0.gpu‚54.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚172.727' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.18175 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¯ëFwá,‚(:ì ÆæêÔ¿õ€ disk.md0.out‚ 1429.29296875 disk.rbd2.in‚0# proc.memory.rssMB‚ 2992.45703125 proc.cpu.threads‚17$ disk./.usageGB‚2713.873275756836 disk.rbd2.out‚ 3.529296875 disk.sda1.in‚0 disk.sdb.in‚ 7.0546875! disk.sda.out‚1482.57080078125! disk.sdb.out‚1482.57080078125) disk./.usagePercent‚75.88103263436838 disk.rbd0.out‚ 3.591796875" disk.sdb2.out‚1482.57080078125 disk.dm-0.in‚ 1118.4375! disk.dm-0.out‚146738.23828125 disk.sda.in‚ 10.7578125) proc.memory.percent‚18.26450824737549 disk.sda2.in‚ 10.7578125 disk.sdb2.in‚ 7.0546875 disk.sda1.out‚0" disk.nvme0n1.out‚ 79602.7265625 disk.md0p1.out‚ 1429.29296875$ memory_percent‚92.10140705108643 disk.md0.in‚17.8125 disk.md0p1.in‚17.8125 disk.nvme0n1.in‚ 619.63671875 cpu‚25.216922810498026 network.sent‚ 2.9239923e+07 network.recv‚ 4.160283e+06" disk.sda2.out‚1482.57080078125 disk.sdb1.out‚0) proc.memory.availableMB‚ 1294.10546875 disk.rbd0.in‚0 disk.nvme1n1.in‚ 498.80078125# disk.nvme1n1.out‚67135.51171875 disk.sdb1.in‚0¨Ùr”â,‚(:‰ ÕæêÔ¬þ¤ÿ gpu.0.gpu‚47.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚160.305' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚40.07625 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¡BÑ–\ã,‚(:Ñ ÕæêÔù–¾€ disk.rbd2.out‚ 3.529296875$ memory_percent‚92.09833145141602' proc.memory.availableMB‚ 1294.609375) disk./.usagePercent‚75.88103210106571 disk.md0p1.in‚17.8125 disk.sdb.out‚ 1484.7578125 disk.md0.out‚ 1431.40625) proc.memory.percent‚18.26450824737549 proc.cpu.threads‚17 disk.sda.in‚ 10.7578125 disk.sda2.in‚ 10.7578125 disk.sda2.out‚ 1484.7578125 disk.nvme1n1.in‚ 498.80078125 disk.dm-0.in‚ 1118.4375% disk./.usageGB‚2713.8732566833496 cpu‚25.216728339623725 disk.sda1.out‚0 network.recv‚ 4.16391e+06 disk.sda.out‚ 1484.7578125" disk.nvme1n1.out‚ 67135.6796875 disk.sdb1.in‚0 disk.md0p1.out‚ 1431.40625 disk.sdb.in‚ 7.0546875 disk.rbd2.in‚0 disk.sdb2.out‚ 1484.7578125 disk.nvme0n1.in‚ 619.63671875 disk.rbd0.in‚0 disk.sdb1.out‚0 disk.md0.in‚17.8125 disk.dm-0.out‚146930.2578125# proc.memory.rssMB‚ 2992.45703125 disk.sdb2.in‚ 7.0546875 disk.sda1.in‚0 network.sent‚ 2.9247444e+07! disk.nvme0n1.out‚ 79794.578125 disk.rbd0.out‚ 3.591796875DZp’ä,‚(:‡ äæêÔ°øÛÿ gpu.0.gpu‚31.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚162.204' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚40.551 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ïåÓóuå,‚(:ê äæêÔè§É€ disk.nvme1n1.in‚ 498.80078125# proc.memory.rssMB‚ 2992.45703125 proc.cpu.threads‚17 network.recv‚ 4.168396e+06 disk.rbd2.in‚0 disk.sdb2.in‚ 7.0546875) proc.memory.percent‚18.26450824737549 disk.sdb1.out‚0 disk.rbd2.out‚ 3.529296875 disk.md0p1.in‚17.8125" disk.sda2.out‚1486.21240234375$ disk./.usageGB‚2713.343505859375 disk.sdb1.in‚0 disk.md0.in‚17.8125 disk.md0p1.out‚ 1432.80859375 disk.nvme0n1.in‚ 619.63671875 disk.md0.out‚ 1432.80859375" disk.sdb2.out‚1486.21240234375 disk.dm-0.out‚147147.8359375 disk.sda.in‚ 10.7578125 disk.rbd0.out‚ 3.591796875 disk.sda1.out‚0 disk.rbd0.in‚0$ memory_percent‚92.09985733032227 disk.sda1.in‚0 disk.dm-0.in‚ 1118.4375# disk.nvme0n1.out‚80011.96484375# disk.nvme1n1.out‚67135.87109375' proc.memory.availableMB‚ 1294.359375 disk.sdb.in‚ 7.0546875! disk.sda.out‚1486.21240234375 disk.sda2.in‚ 10.7578125 cpu‚25.216516622552653 network.sent‚ 2.9253434e+07) disk./.usagePercent‚75.86622004630944! disk.sdb.out‚1486.21240234375¿ù"4æ,/ _wandbruntime‚12993  _runtime‚12993í¢-–Ëç,Å _step‚911 ' lr-AdamW/pg1‚1.7434932214306694e-05 ' lr-AdamW/pg2‚1.7434932214306694e-05 trainer/global_step‚11099 % _timestamp‚1.7885233684795458e+09 _runtime‚12993.950173217L¬„lÐè,Ê ' lr-AdamW/pg1‚1.7434932214306694e-05 ' lr-AdamW/pg2‚1.7434932214306694e-05 trainer/global_step‚11099 % _timestamp‚1.7885233684795458e+09 _runtime‚12993.950173217 _step‚911XTÞ4é,/ _wandbruntime‚12993  _runtime‚12993gõó¦Tê,…P YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚906406201157504 $ _timestamp‚1.788523368480901e+09 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12633025989632 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 216276992 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13193148018176 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 278686228 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚595 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1212 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007241 UFDeviceStatsMonitor.on_train_baΙâ·!tch_start/reserved_bytes.small_pool.peak‚ 1182793728 _step‚912 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚914296615836160 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚66769163 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1022903500059648 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚926929641825792 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 209500703 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12585040870196 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 209501298 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 508252264 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12585938281616 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407872 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚914296597833216 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917410816 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚914296597833216 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13193364295168 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1036096648077824 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12633025989632 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚926929641825792 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚926928724414976 _runtime‚12993.951839145 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955591168 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 229539948 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1187 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚893820245836528 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 508226174 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 278712316 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177047040 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1036096888294912 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚893820262875888 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26088 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686J±¿`(ë,ŠP P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955591168 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407872 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1022903523999744 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚926928724414976 _runtime‚12993.951839145 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955591168 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 229539948 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1187 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚893820245836528 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 508226174 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 278712316 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177047040 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1036096888294912 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚893820262875888 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26088 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚906406201157504 $ _timestamp‚1.788523368480901e+09 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12633025989632 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 216276992 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13193148018176 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 278686228 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚595 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1212 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007241 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 _step‚912 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚914296615836160 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚66769163 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1022903500059648 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚926929641825792 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 209500703 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12585040870196 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 209501298 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 508252264 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12585938281616 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407872 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚914296597833216 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917410816 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚914296597833216 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13193364295168 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1036096648077824 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12633025989632 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚926929641825792 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚914376228128768 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 278710819 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177047040 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 278736933 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚595 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 278710819 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 508296779 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚906484375462170 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 % _timestamp‚1.7885233696612175e+09 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12633239385088 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 209518714 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 209519311 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 235138048 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955591168 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917424128 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚597 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 259078144 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13194527307264 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007241 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 278736933 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 401960960 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 229559846 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 _step‚913 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1036187111449600 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1036187370527744 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177047040 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 trainer/global_step‚11099 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26114 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚914376246131712 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12634138806272 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 229559846 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 142744511 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1022992843220480 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899421184 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 508296779 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚66774800 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12586149531514 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚914376228128768 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 229559844 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚927010384937984 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚927009467513856 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 _runtime‚12995.133538342 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 508270663 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31686 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13194292169216 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12587046943142 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚893898225930656 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 142744509 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1022992819280384 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411628²96Ìî,•N MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1187 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 508270663 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12634138806272 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 229559844 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚893898242970016 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1036187370527744 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177047040 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 trainer/global_step‚11099 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26114 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚914376246131712 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12634138806272 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 229559846 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 142744511 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1022992843220480 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899421184 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 508296779 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚66774800 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12586149531514 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚914376228128768 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 229559844 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚927010384937984 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚927009467513856 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 _runtime‚12995.133538342 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 508270663 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31686 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13194292169216 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12587046943142 F=DeviceStatsMonitor.on_train_batch_end/inactive_€^2Osplit.all.peak‚1212 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚893898225930656 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 142744509 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1022992819280384 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411628 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚914376246131712 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12633239385088 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚906485289913158 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚927010384937984 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26114 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚914376228128768 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 278710819 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177047040 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 278736933 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚595 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 278710819 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 508296779 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚906484375462170 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 % _timestamp‚1.7885233696612175e+09 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12633239385088 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 209518714 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 209519311 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 235138048 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955591168 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917424128 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚597 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 259078144 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13194527307264 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007241 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 278736933 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 401960960 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 229559846 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 _step‚913 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1036187111449600 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚930965893772800 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚930966811183616 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12687781161984 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12688680569856 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450780 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407872 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12687781161984 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955591168 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26088 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1187 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1027369461096448 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚910360652521012 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚897719267582288 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 279945218 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚918278130613760 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚615 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12641384938724 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 210404250 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚910359738070232 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚897719250542928 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 trainer/global_step‚11149 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 143350933 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚930966811183616 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚67053315 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13251405574144 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 279919130 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚617 L¬Š"ÿ,ˆP YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚930966811183616 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12687781161984 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12688680569856 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450780 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407872 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12687781161984 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955591168 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26088 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1187 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1027369461096448 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚910360652521012 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚897719267582288 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 279945218 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚918278130613760 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚615 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12641384938724 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 210404250 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚910359738070232 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚897719250542928 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 trainer/global_step‚11149 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 143350933 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚930966811183616 =6DeviceStatsMonitor.on_train_batchŠÇ²¢ì _start/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚67053315 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13251405574144 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 279919130 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚617 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 510480066 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 210403633 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177047040 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚918278112610816 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 401960960 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214665 _step‚916 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 510453976 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 230534848 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1212 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚930965893772800”ÎS0ï4€-/ _wandbruntime‚13053  _runtime‚13053‘ç'-N YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12688893965312 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 279943721 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚897797230637056 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚13053.266958737 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899421184 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411628 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 510498465 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 510524581 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1187 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955591168 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899421184 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚897797247676416 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 279943721 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917424128 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1027458804257280 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚918357742906368 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 230554746 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 210422267 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 236009472 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12641596188622 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚605 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 143363309 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚67058353 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 % _timestamp‚1.7885234277959795e+09 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177047040 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 401960960 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13252575524864 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚931046636871680 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚67058956 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚918357760909312 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 510498465 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 230554744 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 230554746 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚603 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 279969835 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 279969835 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 210421662 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12689793386496 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26114 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1040711379782144 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚910439741276666 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚910438826825678 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008$Çø'‚-•N VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚67058956 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚918357760909312 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 510498465 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 230554744 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 230554746 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚603 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 279969835 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 279969835 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 210421662 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12689793386496 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26114 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1040711379782144 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚910439741276666 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚910438826825678 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12688893965312 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 279943721 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚897797230637056 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚13053.266958737 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899421184 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411628 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 510498465 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 510524581 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1187 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955591168 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899421184 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚897797247676416 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 279943721 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917424128 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1027458804257280 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚918357742906368 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 230554746 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 210422267 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 236009472 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12641596188622 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚605 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 143363309 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚67058353 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 % _timestamp‚1.7885234277959795e+09 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177047040 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 401960960 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13252575524864 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚931046636871680•:Aj4ƒ-/ _wandbruntime‚13053  _runtime‚13053•™_„-Ù .loss/FBt4/detection_loss‚2.370012477040291 trainer/global_step‚11149 2loss/SRqqt2/assignment_loss‚0.7782672960074969 2loss/SRqqt3/assignment_loss‚1.1635794651024698 1loss/FBt1/assignment_loss‚0.08462263457467618 1loss/SRqqt2/detection_loss‚0.8274926787809186 0loss/SRqqt4/detection_loss‚2.625772163271904 0loss/FRt3/assignment_loss‚2.2144720374151716 2loss/SRqqt4/assignment_loss‚1.4779417744527261 _step‚918 .loss/FRt1/detection_loss‚2.096252230947594 /loss/FRt2/detection_loss‚1.4569036586281772 1loss/SRqqt1/detection_loss‚1.8599422649856199 _runtime‚13053.267913751 0loss/FRt1/assignment_loss‚2.8814689828001936 /loss/FRt2/assignment_loss‚1.251562924339221 2loss/SRqqt1/assignment_loss‚1.2116466723545862 1loss/FBt2/assignment_loss‚0.10978324919273866 .loss/FRt4/detection_loss‚3.044046771937403 epoch‚9 0loss/FRt4/assignment_loss‚3.6305759693014212 2loss/FBt4/assignment_loss‚0.006813193122411576 .loss/FRt3/detection_loss‚2.343891260409078 /loss/FBt2/detection_loss‚0.5638421187298251 /loss/FBt3/detection_loss‚2.2419327166345386 1loss/FBt3/assignment_loss‚0.10960870343736435 1loss/SRqqt3/detection_loss‚1.6673052893265297 $loss/total_loss‚38.3995191969854 /loss/FBt1/detection_loss‚2.3817826641930475 % _timestamp‚1.7885234277977285e+09þŽ?hd…-Þ 1loss/FBt3/assignment_loss‚0.10960870343736435 1loss/SRqqt3/detection_loss‚1.6673052893265297 $loss/total_loss‚38.3995191969854 /loss/FBt1/detection_loss‚2.3817826641930475 % _timestamp‚1.7885234277977285e+09 .loss/FBt4/detection_loss‚2.370012477040291 trainer/global_step‚11149 2loss/SRqqt2/assignment_loss‚0.7782672960074969 2loss/SRqqt3/assignment_loss‚1.1635794651024698 1loss/FBt1/assignment_loss‚0.08462263457467618 1loss/SRqqt2/detection_loss‚0.8274926787809186 0loss/SRqqt4/detection_loss‚2.625772163271904 0loss/FRt3/assignment_loss‚2.2144720374151716 2loss/SRqqt4/assignment_loss‚1.4779417744527261 _step‚918 .loss/FRt1/detection_loss‚2.096252230947594 /loss/FRt2/detection_loss‚1.4569036586281772 1loss/SRqqt1/detection_loss‚1.8599422649856199 _runtime‚13053.267913751 0loss/FRt1/assignment_loss‚2.8814689828001936 /loss/FRt2/assignment_loss‚1.251562924339221 2loss/SRqqt1/assignment_loss‚1.2116466723545862 1loss/FBt2/assignment_loss‚0.10978324919273866 .loss/FRt4/detection_loss‚3.044046771937403 epoch‚9 0loss/FRt4/assignment_loss‚3.6305759693014212 2loss/FBt4/assignment_loss‚0.006813193122411576 .loss/FRt3/detection_loss‚2.343891260409078 /loss/FBt2/detection_loss‚0.5638421187298251 /loss/FBt3/detection_loss‚2.2419327166345386–cº “†-‚(:ˆ ¯çêÔ—¯¾ÿ gpu.0.gpu‚4.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚37.0 gpu.0.powerWatts‚166.207' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚41.55175 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0À·ÞÞo‡-‚(:ä ¯çêÔË„†€ network.recv‚ 4.190325e+06 disk.rbd0.in‚0 disk.rbd0.out‚ 3.666015625) disk./.usagePercent‚75.87408796712192 disk.rbd2.out‚ 3.603515625 disk.sda1.out‚0 disk.md0p1.in‚17.8125$ memory_percent‚92.09961891174316# proc.memory.rssMB‚ 2992.45703125( proc.memory.availableMB‚ 1294.3984375 cpu‚25.21543100407256 network.sent‚ 2.9395308e+07 disk.nvme1n1.in‚ 498.80078125" disk.sdb2.out‚1494.35107421875$ disk./.usageGB‚2713.624900817871" disk.nvme1n1.out‚ 67137.0390625 disk.sdb1.in‚0 disk.md0p1.out‚ 1440.6171875 disk.sda.in‚ 10.7578125 disk.sda2.in‚ 10.7578125 disk.md0.in‚17.8125 disk.sda1.in‚0 disk.dm-0.in‚ 1118.4375" disk.nvme0n1.out‚ 81167.8046875 disk.sdb1.out‚0 disk.md0.out‚ 1440.6171875 disk.rbd2.in‚0 disk.sdb2.in‚ 7.0546875! disk.sdb.out‚1494.35107421875! disk.sda.out‚1494.35107421875" disk.sda2.out‚1494.35107421875) proc.memory.percent‚18.26450824737549 disk.dm-0.out‚ 148304.84375 disk.nvme0n1.in‚ 619.63671875 disk.sdb.in‚ 7.0546875 proc.cpu.threads‚17–ˆ³8”ˆ-‚(:‰ ¾çêÔá“øý gpu.0.gpu‚53.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚171.501' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.87525 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ØV®i‰-‚(:Þ ¾çêÔÕýœÿ network.recv‚ 4.196386e+06 disk.md0.out‚ 1442.38671875 disk.rbd2.out‚ 3.70556640625 disk.sdb2.in‚ 7.0546875) proc.memory.percent‚18.26450824737549 disk.sdb.in‚ 7.0546875 disk.sda1.out‚0 disk.dm-0.in‚ 1118.4375 disk.rbd0.in‚0 disk.sdb1.out‚0" proc.memory.availableMB‚1294.5# proc.memory.rssMB‚ 2992.45703125 disk.sda.in‚ 10.7578125 disk.sdb1.in‚0 disk.md0.in‚17.8125 disk.nvme0n1.in‚ 619.63671875) disk./.usagePercent‚75.88109108433999 disk.sda1.in‚0 disk.md0p1.in‚17.8125 disk.md0p1.out‚ 1442.38671875 disk.sda.out‚1496.173828125 proc.cpu.threads‚17 disk.sdb.out‚1496.173828125 disk.rbd2.in‚0# memory_percent‚92.0989990234375# disk.nvme0n1.out‚81443.60546875% disk./.usageGB‚2713.8753662109375 disk.sda2.in‚ 10.7578125 disk.nvme1n1.in‚ 498.80078125 disk.sdb2.out‚1496.173828125! disk.dm-0.out‚148580.86328125 disk.rbd0.out‚ 3.666015625" disk.nvme1n1.out‚ 67137.2578125 cpu‚25.21520681839337 network.sent‚ 2.9492747e+07 disk.sda2.out‚1496.173828125Ø攊-‚(:‰ ÍçêÔùŽÕý gpu.0.gpu‚36.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚137.083' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚34.27075 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÊÉlq‹-‚(:æ ÍçêÔϹõþ) proc.memory.percent‚18.26450824737549 network.sent‚ 2.9495801e+07 disk.rbd0.out‚ 3.7451171875 disk.rbd2.out‚ 3.70556640625 disk.md0p1.in‚17.8125) proc.memory.availableMB‚ 1294.85546875# disk.nvme0n1.out‚81629.27734375 disk.sda.in‚ 10.7578125 disk.nvme1n1.in‚ 498.80078125 cpu‚25.215019186188012) disk./.usagePercent‚75.88106100606993 disk.nvme0n1.in‚ 619.63671875 disk.sda.out‚1498.0087890625# disk.nvme1n1.out‚67137.80859375% disk./.usageGB‚2713.8742904663086 disk.sdb.out‚1498.0087890625! disk.sda2.out‚1498.0087890625 disk.dm-0.in‚ 1118.4375 disk.md0p1.out‚ 1444.15625# proc.memory.rssMB‚ 2992.45703125 proc.cpu.threads‚17 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.md0.out‚ 1444.15625 disk.rbd2.in‚0! disk.sdb2.out‚1498.0087890625$ memory_percent‚92.09682941436768 network.recv‚ 4.199684e+06 disk.sda2.in‚ 10.7578125 disk.md0.in‚17.8125 disk.dm-0.out‚148767.0859375 disk.sdb.in‚ 7.0546875 disk.rbd0.in‚0 disk.sdb2.in‚ 7.0546875 disk.sda1.in‚0 disk.sda1.out‚0[’ìjsŒ-‚(:è ÜçêÔ‰ÅÝÿ disk.md0.in‚17.8125 disk.rbd2.out‚ 3.7451171875 disk.sda1.in‚0 disk.sdb.in‚ 7.0546875 disk.sdb.out‚1499.7783203125 cpu‚25.21478939683086 proc.cpu.threads‚17 disk.md0p1.in‚17.8125 disk.sda.in‚ 10.7578125# proc.memory.rssMB‚ 2992.45703125 network.recv‚ 4.204237e+06) disk./.usagePercent‚75.88106100606993% disk./.usageGB‚2713.8742904663086# disk.nvme1n1.out‚67138.01171875 disk.md0.out‚ 1445.85546875 disk.sda1.out‚0 disk.md0p1.out‚ 1445.85546875 disk.nvme1n1.in‚ 498.80078125! disk.sda2.out‚1499.7783203125) proc.memory.percent‚18.26450824737549 disk.rbd0.out‚ 3.7451171875 disk.sdb1.out‚0 disk.nvme0n1.in‚ 619.63671875 disk.sda.out‚1499.7783203125 disk.sda2.in‚ 10.7578125$ memory_percent‚92.09764003753662) proc.memory.availableMB‚ 1294.72265625 disk.dm-0.out‚ 148996.40625# disk.nvme0n1.out‚81858.39453125 disk.sdb1.in‚0 disk.rbd2.in‚0 disk.sdb2.in‚ 7.0546875! disk.sdb2.out‚1499.7783203125 disk.dm-0.in‚ 1118.4375 network.sent‚ 2.9501804e+07 disk.rbd0.in‚0Ëaµ…”-‚(:‰ ÜçêÔÅèÝÿ gpu.0.gpu‚34.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚37.0 gpu.0.powerWatts‚144.821' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚36.20525 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ö ™4Ž-/ _wandbruntime‚13110  _runtime‚13110ÅIGãFdÉ-Å % _timestamp‚1.7885234850836954e+09 _runtime‚13110.554133979 _step‚919 ' lr-AdamW/pg1‚1.4112269132292371e-05 ' lr-AdamW/pg2‚1.4112269132292371e-05 trainer/global_step‚11199'|šÐ-Ê _runtime‚13110.554133979 _step‚919 ' lr-AdamW/pg1‚1.4112269132292371e-05 ' lr-AdamW/pg2‚1.4112269132292371e-05 trainer/global_step‚11199 % _timestamp‚1.7885234850836954e+09—¤‘p4‘-/ _wandbruntime‚13110  _runtime‚13110k·º (’-†P HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410816 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚601 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 trainer/global_step‚11199 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 512681778 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 231529746 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 281178120 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚901618255249328 % _timestamp‚1.7885234850849571e+09 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1031835446073344 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 207888384 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1045144708061184 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1045144939889664 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1212 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 231828480 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 281178120 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚67338182 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 401960960 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214665 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚922259627388416 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚922259645391360 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚935003063130624 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26088 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917410816 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 231529748 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12744335150080 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 143969735 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177047040 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 143969733 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1187 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12743435742208 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚922259627388416 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚67337583 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚914314189433740 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955591168 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚922259645391360 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12743435742208 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 231529748 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955591168 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007241 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407872 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚599 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _step‚920 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚935003980541440 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 211307917 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13309493816320 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450780 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚901618272288688 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚935003063130624 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12696831595832 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13309285927936 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 281152032 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411420 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12695934184412 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26088 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407872 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 211307316 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚935003980541440 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 281152032 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12744335150080Ø»Šá(“-‹P JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917410816 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 231529748 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12744335150080 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 143969735 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177047040 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 143969733 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1187 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12743435742208 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚922259627388416 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚67337583 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚914314189433740 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955591168 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚922259645391360 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12743435742208 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 231529748 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955591168 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007241 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407872 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚599 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _step‚920 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚935003980541440 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 211307917 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13309493816320 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450780 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚901618272288688 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚935003063130624 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12696831595832 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13309285927936 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 281152032 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411420 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12695934184412 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26088 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407872 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 211307316 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚935003980541440 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 281152032 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12744335150080 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410816 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚601 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 trainer/global_step‚11199 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 512681778 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 231529746 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 281178120 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚901618255249328 % _timestamp‚1.7885234850849571e+09 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1031835446073344 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 207888384 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1045144708061184 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1045144939889664 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1212 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 231828480 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 281178120 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚67338182 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 401960960 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214665 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚922259627388416 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚922259645391360 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚935003063130624 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26088˜M&kÌ4”-/ _wandbruntime‚13111  _runtime‚13111ÙÕ8y'•-N FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 231549646 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1045235183224320 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 401960960 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411628 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12744548545536 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1031924741353984 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 224652288 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 211325919 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚618 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1031924765294080 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12745447966720 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 231549646 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31686 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 trainer/global_step‚11199 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 231549644 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚922339275686912 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 248592384 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚901696252382816 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 281176623 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955591168 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚922339257683968 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 512726267 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 $ _timestamp‚1.788523486106935e+09 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚922339257683968 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 512752383 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 281176623 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚914393278189186 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12697940257358 _step‚921 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚67343192 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007241 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899421184 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899421184 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚901696235343456 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 281202737 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26114 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917424128 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 211325301 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 231549644 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 143982111 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1187 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚935084723653632 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12744548545536 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12697042845730 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0“ó7Š•–-”N VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411628 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12744548545536 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1031924741353984 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 224652288 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 211325919 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚618 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1031924765294080 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12745447966720 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 SEDevic•Ò¦ï… eStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 231549646 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31686 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 trainer/global_step‚11199 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 231549644 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚922339275686912 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 248592384 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚901696252382816 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 281176623 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955591168 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚922339257683968 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 512726267 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 $ _timestamp‚1.788523486106935e+09 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚922339257683968 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 512752383 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 281176623 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚914393278189186 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12697940257358 _step‚921 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚67343192 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007241 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899421184 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899421184 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚901696235343456 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 281202737 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26114 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917424128 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 211325301 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 231549644 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 143982111 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1187 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚935084723653632 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12744548545536 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12697042845730 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 231549646 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1045235183224320 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 401960960™¹4YI4—-/ _wandbruntime‚13111  _runtime‚131116.ÚZ˜-Ô 0loss/FRt2/assignment_loss‚1.5678462898522092 0loss/FRt3/assignment_loss‚2.5118592052083266 0loss/FBt2/detection_loss‚0.49809715536809185 %loss/total_loss‚38.05694972008654 epoch‚9 /loss/FRt4/assignment_loss‚3.465686187148094 1loss/SRqqt4/assignment_loss‚1.720760677975637 1loss/FBt3/assignment_loss‚0.18076670041027398 .loss/FRt2/detection_loss‚1.586050234800368 1loss/SRqqt2/detection_loss‚0.9489972559735179 .loss/FBt4/detection_loss‚2.221150783697764 1loss/FBt2/assignment_loss‚0.21414884719306615 1loss/FBt4/assignment_loss‚0.11588640580378827 1loss/SRqqt1/detection_loss‚1.4652902201050892 .loss/FBt3/detection_loss‚2.558268401771784 trainer/global_step‚11199 % _timestamp‚1.7885234861102421e+09 1loss/SRqqt3/detection_loss‚1.5314595150028139 _runtime‚13111.580500292 /loss/FRt1/assignment_loss‚1.974512558716994 2loss/SRqqt1/assignment_loss‚1.1734813482228976 1loss/SRqqt2/assignment_loss‚1.169802224835647 0loss/FBt1/assignment_loss‚0.2938079734412824 0loss/SRqqt4/detection_loss‚2.709122993327953 .loss/FBt1/detection_loss‚2.210059329867363 _step‚922 1loss/SRqqt3/assignment_loss‚1.087294429481188 /loss/FRt1/detection_loss‚1.8505822393669864 /loss/FRt3/detection_loss‚2.0216070533190904 .loss/FRt4/detection_loss‚2.980411689196314ªn<@_™-Ù /loss/FRt1/assignment_loss‚1.974512558716994 2loss/SRqqt1/assignment_loss‚1.1734813482228976 1loss/SRqqt2/assignment_loss‚1.169802224835647 0loss/FBt1/assignment_loss‚0.2938079734412824 0loss/SRqqt4/detection_loss‚2.709122993327953 .loss/FBt1/detection_loss‚2.210059329867363 _step‚922 1loss/SRqqt3/assignment_loss‚1.087294429481188 /loss/FRt1/detection_loss‚1.8505822393669864 /loss/FRt3/detection_loss‚2.0216070533190904 .loss/FRt4/detection_loss‚2.980411689196314 0loss/FRt2/assignment_loss‚1.5678462898522092 0loss/FRt3/assignment_loss‚2.5118592052083266 0loss/FBt2/detection_loss‚0.49809715536809185 %loss/total_loss‚38.05694972008654 epoch‚9 /loss/FRt4/assignment_loss‚3.465686187148094 1loss/SRqqt4/assignment_loss‚1.720760677975637 1loss/FBt3/assignment_loss‚0.18076670041027398 .loss/FRt2/detection_loss‚1.586050234800368 1loss/SRqqt2/detection_loss‚0.9489972559735179 .loss/FBt4/detection_loss‚2.221150783697764 1loss/FBt2/assignment_loss‚0.21414884719306615 1loss/FBt4/assignment_loss‚0.11588640580378827 1loss/SRqqt1/detection_loss‚1.4652902201050892 .loss/FBt3/detection_loss‚2.558268401771784 trainer/global_step‚11199 % _timestamp‚1.7885234861102421e+09 1loss/SRqqt3/detection_loss‚1.5314595150028139 _runtime‚13111.580500292šE*+n’š-‚(:‡ ëçêÔïõÓý gpu.0.gpu‚49.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚175.844' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.961 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 T²Ir›-‚(:ç ëçêÔ܇ÿ disk.nvme1n1.in‚ 498.80078125 disk.dm-0.in‚ 1118.4375 disk.sdb.out‚1501.748046875 network.recv‚ 4.207739e+06) disk./.usagePercent‚75.88106228599631 disk.sda.out‚1501.748046875# disk.nvme1n1.out‚67138.16015625 disk.rbd2.in‚0 disk.sdb2.out‚1501.748046875 disk.sda1.in‚0 disk.sdb1.out‚0$ disk./.usageGB‚2713.874336242676 disk.rbd0.in‚0 disk.sda2.out‚1501.748046875 disk.sdb.in‚ 7.0546875 disk.rbd2.out‚ 3.7451171875! disk.dm-0.out‚149297.69921875 disk.md0p1.in‚17.8125 disk.nvme0n1.in‚ 619.63671875" disk.nvme0n1.out‚ 82159.5390625 disk.sda1.out‚0$ memory_percent‚92.09921360015869) proc.memory.availableMB‚ 1294.46484375) proc.memory.percent‚18.26450824737549 proc.cpu.threads‚17 disk.md0.out‚ 1447.75390625 disk.sdb2.in‚ 7.0546875 disk.md0p1.out‚ 1447.75390625 network.sent‚ 2.9507445e+07 disk.sda.in‚ 10.7578125 disk.sda2.in‚ 10.7578125 disk.rbd0.out‚ 3.80810546875 disk.sdb1.in‚0 disk.md0.in‚17.8125# proc.memory.rssMB‚ 2992.45703125 cpu‚25.214585148720946ïÓr­’œ-‚(:‡ úçêÔá½ãý gpu.0.gpu‚39.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚175.512' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.878 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0-1€Àj-‚(:ß úçêÔçûÕþ disk.md0p1.out‚ 1449.09375 disk.nvme0n1.in‚ 619.63671875# disk.nvme0n1.out‚82485.62890625 disk.sda.out‚ 1503.15234375 disk.sda2.out‚ 1503.15234375 disk.sdb1.out‚0 disk.rbd2.in‚0 disk.md0p1.in‚17.8125# disk.nvme1n1.out‚67138.37890625% disk./.usageGB‚2713.8734741210938 disk.sdb2.in‚ 7.0546875 disk.sdb2.out‚ 1503.15234375 disk.nvme1n1.in‚ 498.80078125 disk.md0.out‚ 1449.09375) proc.memory.availableMB‚ 1294.46484375 proc.cpu.threads‚17) disk./.usagePercent‚75.88103818071605 disk.sdb.out‚ 1503.15234375 disk.sda.in‚ 10.7578125 disk.rbd0.out‚ 3.80810546875 disk.md0.in‚17.8125 network.recv‚ 4.213747e+06 disk.rbd2.out‚ 3.80810546875# proc.memory.rssMB‚ 2992.45703125 cpu‚25.214380162011256 disk.dm-0.in‚ 1118.4375 disk.dm-0.out‚149624.0078125 disk.sda2.in‚ 10.7578125 disk.rbd0.in‚0 disk.sdb1.in‚0 network.sent‚ 2.9614871e+07 disk.sda1.in‚0 disk.sda1.out‚0 disk.sdb.in‚ 7.0546875$ memory_percent‚92.09921360015869) proc.memory.percent‚18.26450824737549Yåi攞-‚(:‰ ‰èêÔð«åý gpu.0.gpu‚40.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚37.0 gpu.0.powerWatts‚117.193' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚29.29825 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0<&ºôhŸ-‚(:Ý ‰èêÔ†ðèþ disk.sda.in‚ 10.7578125 disk.md0.in‚17.8125 cpu‚25.214173528296413 disk.sda2.in‚ 10.7578125 disk.sda2.out‚ 1504.63671875 disk.rbd0.in‚0 disk.rbd0.out‚ 3.87109375! disk.dm-0.out‚149861.61328125 disk.md0p1.in‚17.8125 disk.nvme0n1.in‚ 619.63671875! disk.nvme0n1.out‚ 82722.828125 network.recv‚ 4.217214e+06 disk.sdb1.out‚0 disk.sdb.in‚ 7.0546875 proc.cpu.threads‚17 network.sent‚ 2.9620138e+07# disk.nvme1n1.out‚67138.78515625 disk.sda1.out‚0$ memory_percent‚92.09766387939453& proc.memory.availableMB‚ 1294.71875 disk.sdb2.in‚ 7.0546875) disk./.usagePercent‚75.86622612595978$ disk./.usageGB‚2713.343723297119 disk.sda.out‚ 1504.63671875 disk.nvme1n1.in‚ 498.80078125 disk.sdb1.in‚0 disk.md0.out‚ 1450.51953125 disk.rbd2.out‚ 3.80810546875 disk.sda1.in‚0 disk.rbd2.in‚0 disk.sdb2.out‚ 1504.63671875 disk.dm-0.in‚ 1118.4375 disk.md0p1.out‚ 1450.51953125# proc.memory.rssMB‚ 2992.45703125) proc.memory.percent‚18.26450824737549 disk.sdb.out‚ 1504.63671875E|…4 -/ _wandbruntime‚13168  _runtime‚13168€©tÈ¡- & lr-AdamW/pg1‚1.258041514330103e-05 & lr-AdamW/pg2‚1.258041514330103e-05 trainer/global_step‚11249 $ _timestamp‚1.788523543430521e+09 _runtime‚13168.901020388 _step‚923(µvÍ¢-Ç & lr-AdamW/pg2‚1.258041514330103e-05 trainer/global_step‚11249 $ _timestamp‚1.788523543430521e+09 _runtime‚13168.901020388 _step‚923 & lr-AdamW/pg1‚1.258041514330103e-05›±n04£-/ _wandbruntime‚13168  _runtime‚13168Ý'ì (¤-…P F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955591168 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 144588535 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚607 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚67623187 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 216276992 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 232524646 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12799989730304 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955591168 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1187 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407872 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410816 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407872 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚939041149899264 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12799090322432 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 144588533 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚67622582 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1049668913372160 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _runtime‚13168.902869157 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚926241160168960 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13367289985024 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13367506262016 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12751380841520 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 282384934 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917410816 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚939040232488448 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 401960960 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚905517259955728 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 212211722 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1212 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 212211115 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚918269555248028 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 514909580 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚939041149899264 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚926241142166016 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 282411022 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1049668673155072 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1036301407110144 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 282411022 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12799090322432 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚905517276995088 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 232524646 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26088 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 282384934 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214665 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚918268640797248 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 trainer/global_step‚11249 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚939040232488448 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 232524648 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1036301383170048 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450780 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177047040 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚926241160168960 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚605 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 232524648 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12752278252940 _step‚924 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12799989730304 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 240217088 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411420ƒW¥-ŠP D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450780 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177047040 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚926241160168960 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚605 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 232524648 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12752278252940 _step‚924 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12799989730304 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 240217088 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411420 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955591168 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 144588535 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚607 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚67623187 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 216276992 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 232524646 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12799989730304 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955591168 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1187 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407872 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410816 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407872 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚939041149899264 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12799090322432 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 144588533 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚67622582 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1049668913372160 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _runtime‚13168.902869157 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚926241160168960 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13367289985024 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13367506262016 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12751380841520 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 282384934 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917410816 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚939040232488448 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batcôNþöh_start/inactive_split_bytes.all.peak‚ 401960960 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚905517259955728 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 212211722 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1212 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 212211115 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚918269555248028 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 514909580 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚939041149899264 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚926241142166016 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 282411022 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1049668673155072 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1036301407110144 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 282411022 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12799090322432 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚905517276995088 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 232524646 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26088 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 282384934 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214665 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚918268640797248 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 trainer/global_step‚11249 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚939040232488448 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 232524648 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1036301383170048œäìʼ4¦-/ _wandbruntime‚13169  _runtime‚13169|Ðk'§-N H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚612 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31686 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 228846592 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚905595257089216 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 401960960 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13368671363072 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26114 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 282409525 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899421184 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚926320772461568 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _runtime‚13169.913694455 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12752489502838 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177047040 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚67628162 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1049759397693952 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚918347729552694 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007241 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12800203125760 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 514980185 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 212229071 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177047040 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450988 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12753386914466 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26114 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1187 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 514954069 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411628 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 _step‚925.°:'¨-•N M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 282409525 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚939121893011456 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚939121893011456 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _runtime‚13169.913694455 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12752489502838 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177047040 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚67628162 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1049759397693952 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚918347729552694 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007241 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12800203125760 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 514980185 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 212229071 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177047040 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450988 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12753386914466 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26114 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1187 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 514954069 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411628 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 _step‚925 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚612 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31686 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 228846592 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚905595257089216 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 401960960 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13368671363072 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26114 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 282409525 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899421184 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚926320772461568 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 B¨‘ë˵-Å ' lr-AdamW/pg2‚1.1135311174435932e-05 trainer/global_step‚11299 % _timestamp‚1.7885236016342642e+09 _runtime‚13227.104651382 _step‚927 ' lr-AdamW/pg1‚1.1135311174435932e-05E[k¶Ð¶-Ê ' lr-AdamW/pg1‚1.1135311174435932e-05 ' lr-AdamW/pg2‚1.1135311174435932e-05 trainer/global_step‚11299 % _timestamp‚1.7885236016342642e+09 _runtime‚13227.104651382 _step‚927Ÿ[î4·-/ _wandbruntime‚13227  _runtime‚13227ÙÌsp (¸-†P F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007241 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1040767368146944 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13425278365696 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 283617836 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 213114641 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 254897152 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 _runtime‚13227.105829557 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12855644310528 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚611 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚909416264662128 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177047040 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚613 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚67907308 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12854744902656 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚943077401846272 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955591168 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 _step‚928 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407872 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚930222674946560 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450780 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1054192877469696 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 283617836 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚930222656943616 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12855644310528 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407872 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13425509322752 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚943078319257088 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 213115254 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚922224006611536 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚930222674946560 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26088 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 145207333 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 401960960 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955591168 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12854744902656 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 trainer/global_step‚11299 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 283643924 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410816 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚67907919 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 Äýð(¹-‹P F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177047040 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚613 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚67907308 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12854744902656 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚943077401846272 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955591168 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 _step‚928 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407872 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚930222674946560 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450780 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1054192877469696 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 283617836 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚930222656943616 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12855644310528 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407872 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13425509322752 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚943078319257088 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 213115254 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚922224006611536 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚930222674946560 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26088 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 145207333 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 401960960 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955591168 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12854744902656 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 trainer/global_step‚11299 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 283643924 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410816 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚67907919 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007241 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1040767368146944 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13425278365696 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 283617836 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 213114641 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 254897152 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 _runtime‚13227.105829557 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12855644310528 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚611 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚909416264662128 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 dÎÖt4º-/ _wandbruntime‚13228  _runtime‚13228ÊÔ!|'»-N L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1040856663427584 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12808833571574 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚943159062369280 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚67913641 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 220457984 _runtime‚13228.26108567 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚597 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 401960960 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚909494244756256 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 517181871 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚929 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1054283354746368 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 233539446 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 283668541 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12856757127168 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12807936159946 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12855857705984 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 283668541 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26114 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1054283110348288 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13426446920704 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 517207987 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚909494261795616 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 213133352 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1040856687367680 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 trainer/global_step‚11299 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13426667378688 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 145219709 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26114 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31686 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 233539444 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177047040 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚599 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚943159062369280 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚930302287239168 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411628 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 517181871 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 213132753 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚67913044 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 244398080 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899421184 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 145219711 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7885236027899792e+09 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 517207987 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177047040 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 233539444 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955591168 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450988®W`5‚¼-”N % _timestamp‚1.7885236027899792e+09 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 517207987 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177047040 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 233539444 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955591168 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450988 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1040856663427584 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12808833571574 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚943159062369280 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚67913641 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 220457984 _runtime‚13228.26108567 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small“½"/˜"_pool.current‚597 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 401960960 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚909494244756256 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 517181871 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚929 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1054283354746368 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 233539446 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 283668541 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12856757127168 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12807936159946 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12855857705984 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 283668541 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26114 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1054283110348288 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13426446920704 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 517207987 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚909494261795616 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 213133352 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1040856687367680 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 trainer/global_step‚11299 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13426667378688 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 145219709 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26114 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31686 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 233539444 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177047040 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚599 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚943159062369280 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚930302287239168 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411628 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 517181871 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 213132753 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚67913044 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 244398080 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899421184 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 145219711 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0¡^ýo4½-/ _wandbruntime‚13228  _runtime‚13228Ò#ñÐ]¾-× 2loss/SRqqt3/assignment_loss‚1.1916802752689988 2loss/SRqqt4/assignment_loss‚1.6863987203155244 1loss/FBt1/assignment_loss‚0.23887334279404135 $loss/total_loss‚37.1002197321363 1loss/SRqqt1/assignment_loss‚1.194886169299997 1loss/FBt2/assignment_loss‚0.11003227658721691 /loss/FRt2/detection_loss‚1.1683304163537198 .loss/FRt3/detection_loss‚2.469370495080948 .loss/FRt4/detection_loss‚3.331983576650205 /loss/FRt4/assignment_loss‚2.779576247153075 1loss/SRqqt1/detection_loss‚1.5797986701323554 /loss/FBt4/detection_loss‚2.3875409588217735 _runtime‚13228.262122085 0loss/SRqqt3/detection_loss‚1.618701682809521 0loss/SRqqt4/detection_loss‚2.387498824085508 0loss/FRt2/assignment_loss‚0.8814745285818654 1loss/FBt3/assignment_loss‚0.04756638128274062 1loss/SRqqt2/detection_loss‚0.7674181452287095 /loss/FBt1/detection_loss‚2.6046400368213654 /loss/FBt2/detection_loss‚0.5590087534640642 trainer/global_step‚11299 0loss/FRt1/assignment_loss‚2.3192741414333913 2loss/SRqqt2/assignment_loss‚1.0164047900853412 /loss/FRt1/detection_loss‚1.9512492144642741 epoch‚9 _step‚930 0loss/FRt3/assignment_loss‚2.3913698983192444 1loss/FBt4/assignment_loss‚0.16433007572470615 .loss/FBt3/detection_loss‚2.252812111377716 % _timestamp‚1.7885236027919652e+09ÅíÈvb¿-Ü 2loss/SRqqt3/assignment_loss‚1.1916802752689988 2loss/SRqqt4/assignment_loss‚1.6863987203155244 1loss/FBt1/assignment_loss‚0.23887334279404135 $loss/total_loss‚37.1002197321363 1loss/SRqqt1/assignment_loss‚1.194886169299997 1loss/FBt2/assignment_loss‚0.11003227658721691 /loss/FRt2/detection_loss‚1.1683304163537198 .loss/FRt3/detection_loss‚2.469370495080948 .loss/FRt4/detection_loss‚3.331983576650205 /loss/FRt4/assignment_loss‚2.779576247153075 1loss/SRqqt1/detection_loss‚1.5797986701323554 /loss/FBt4/detection_loss‚2.3875409588217735 _runtime‚13228.262122085 0loss/SRqqt3/detection_loss‚1.618701682809521 0loss/SRqqt4/detection_loss‚2.387498824085508 0loss/FRt2/assignment_loss‚0.8814745285818654 1loss/FBt3/assignment_loss‚0.04756638128274062 1loss/SRqqt2/detection_loss‚0.7674181452287095 /loss/FBt1/detection_loss‚2.6046400368213654 /loss/FBt2/detection_loss‚0.5590087534640642 trainer/global_step‚11299 0loss/FRt1/assignment_loss‚2.3192741414333913 2loss/SRqqt2/assignment_loss‚1.0164047900853412 /loss/FRt1/detection_loss‚1.9512492144642741 epoch‚9 _step‚930 0loss/FRt3/assignment_loss‚2.3913698983192444 1loss/FBt4/assignment_loss‚0.16433007572470615 .loss/FBt3/detection_loss‚2.252812111377716 % _timestamp‚1.7885236027919652e+09¢égË&”À-‚(:‰ ÔèêÔðÖ°ÿ gpu.0.gpu‚29.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚164.989' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚41.24725 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚09aMiÁ-‚(:Þ ÔèêÔ š‹# disk.nvme0n1.out‚83756.16015625 disk.rbd2.in‚0 disk.sda.out‚1513.4423828125 disk.sdb1.out‚0$ memory_percent‚92.09556579589844! disk.sdb2.out‚1513.4423828125 disk.md0p1.out‚ 1459.00390625 disk.sdb.out‚1513.4423828125 disk.sda2.in‚ 10.7578125 disk.sda1.in‚0 disk.sda1.out‚0 network.sent‚ 2.9764413e+07( disk./.usagePercent‚75.8740795409399 disk.nvme1n1.in‚ 498.80078125# proc.memory.rssMB‚ 2992.45703125) proc.memory.percent‚18.26450824737549 proc.cpu.threads‚17! disk.dm-0.out‚150896.72265625$ disk./.usageGB‚2713.624599456787 disk.rbd2.out‚ 3.87109375 disk.nvme0n1.in‚ 619.63671875 disk.sda.in‚ 10.7578125 disk.sdb1.in‚0 disk.sdb2.in‚ 7.0546875! disk.sda2.out‚1513.4423828125 disk.nvme1n1.out‚ 67140.5625% proc.memory.availableMB‚ 1295.0625 disk.sdb.in‚ 7.0546875 disk.rbd0.out‚ 3.87109375 disk.dm-0.in‚ 1118.4375 network.recv‚ 4.239703e+06 disk.md0.in‚17.8125 disk.md0.out‚ 1459.00390625 disk.rbd0.in‚0 cpu‚25.213084728092458 disk.md0p1.in‚17.8125æ\Ô”Â-‚(:‰ ãèêÔ¢§¬ÿ gpu.0.gpu‚21.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚163.257' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚40.81425 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ôϵImÃ-‚(:â ãèêÔ²üË€! disk.sda.out‚1514.95947265625( proc.memory.availableMB‚ 1294.8203125) proc.memory.percent‚18.26450824737549 disk.sda2.in‚ 10.7578125 disk.rbd0.in‚0 disk.md0.in‚17.8125 disk.rbd0.out‚ 3.87109375 disk.nvme1n1.in‚ 498.80078125 disk.sdb1.out‚0 disk.md0.out‚ 1460.46875" disk.sdb2.out‚1514.95947265625 disk.sda1.in‚0 proc.cpu.threads‚17 network.sent‚ 2.9859274e+07 network.recv‚ 4.244991e+06 disk.dm-0.in‚ 1118.4375! disk.dm-0.out‚151113.00390625 disk.nvme0n1.in‚ 619.63671875$ memory_percent‚92.09704399108887 cpu‚25.21288353926623) disk./.usagePercent‚75.88109652402711$ disk./.usageGB‚2713.875560760498" disk.nvme1n1.out‚ 67140.7421875 disk.rbd2.in‚0! disk.sdb.out‚1514.95947265625# disk.nvme0n1.out‚83972.26171875# proc.memory.rssMB‚ 2992.45703125" disk.sda2.out‚1514.95947265625 disk.md0p1.in‚17.8125 disk.sdb1.in‚0 disk.rbd2.out‚ 3.87109375 disk.sda1.out‚0 disk.sdb.in‚ 7.0546875 disk.sda.in‚ 10.7578125 disk.sdb2.in‚ 7.0546875 disk.md0p1.out‚ 1460.46875´ZQg‘Ä-‚(:† òèêÔÐîèý gpu.0.gpu‚50.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚37.0 gpu.0.powerWatts‚114.74' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.685 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0-a€ûcÅ-‚(:Ø òèêÔ„ñÌþ disk.sdb.out‚ 1516.45703125 disk.sda.out‚ 1516.45703125 disk.sdb1.out‚0 cpu‚25.212669873362994) disk./.usagePercent‚75.88104511365063 disk.sdb1.in‚0 disk.md0p1.out‚ 1461.8984375 disk.sda2.out‚ 1516.45703125# disk.nvme1n1.out‚67140.90234375 disk.md0.out‚ 1461.8984375 disk.sda1.out‚0" disk.nvme0n1.out‚ 84269.9921875$ memory_percent‚92.09861755371094# proc.memory.rssMB‚ 2992.45703125 network.recv‚ 4.250029e+06$ disk./.usageGB‚2713.873722076416 disk.sdb.in‚ 7.0546875 disk.sda.in‚ 10.7578125 disk.md0.in‚17.8125 disk.rbd2.out‚ 3.87109375 disk.sda1.in‚0 disk.dm-0.in‚ 1118.4375 network.sent‚ 2.9865445e+07 disk.rbd0.in‚0 disk.nvme1n1.in‚ 498.80078125 disk.rbd2.in‚0 disk.sdb2.out‚ 1516.45703125! disk.dm-0.out‚151410.89453125 disk.md0p1.in‚17.8125 disk.nvme0n1.in‚ 619.63671875 disk.rbd0.out‚ 3.87109375% proc.memory.availableMB‚ 1294.5625) proc.memory.percent‚18.26450824737549 proc.cpu.threads‚17 disk.sda2.in‚ 10.7578125 disk.sdb2.in‚ 7.0546875Œ™Ë‘Æ-‚(:† éêÔŠâòý gpu.0.gpu‚37.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚37.0 gpu.0.powerWatts‚114.74' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.685 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0åÔ¼nÇ-‚(:ã éêÔÃÎÿþ disk.sda1.out‚0 disk.md0p1.in‚17.8125 disk.sdb.in‚ 7.0546875 proc.cpu.threads‚17 disk.sda1.in‚0 disk.nvme0n1.in‚ 619.63671875 disk.sda.in‚ 10.7578125 disk.rbd0.out‚ 3.87109375 disk.sdb1.in‚0 disk.md0.in‚17.8125) proc.memory.percent‚18.26450824737549 network.sent‚ 2.9870752e+07" disk.sda2.out‚1518.41552734375 disk.dm-0.in‚ 1118.4375 disk.dm-0.out‚ 151676.453125 disk.nvme1n1.in‚ 498.80078125 disk.md0.out‚ 1463.78515625 disk.rbd2.out‚ 3.87109375% proc.memory.availableMB‚ 1294.3125# proc.memory.rssMB‚ 2992.45703125 network.recv‚ 4.253509e+06" disk.sdb2.out‚1518.41552734375 disk.sda2.in‚ 10.7578125 disk.rbd2.in‚0 cpu‚25.212484837070544 disk.md0p1.out‚ 1463.78515625" disk.nvme0n1.out‚ 84535.1796875) disk./.usagePercent‚75.88104351374265 disk.sdb2.in‚ 7.0546875! disk.sda.out‚1518.41552734375 disk.rbd0.in‚0$ memory_percent‚92.10014343261719$ disk./.usageGB‚2713.873664855957! disk.sdb.out‚1518.41552734375" disk.nvme1n1.out‚ 67141.2734375 disk.sdb1.out‚0ߊé4È-/ _wandbruntime‚13285  _runtime‚13285e‰ÈÉ- & lr-AdamW/pg1‚9.777258383198278e-06 & lr-AdamW/pg2‚9.777258383198278e-06 trainer/global_step‚11349 $ _timestamp‚1.788523660067546e+09 _runtime‚13285.537943303 _step‚931{à3ÍÊ-Ç trainer/global_step‚11349 $ _timestamp‚1.788523660067546e+09 _runtime‚13285.537943303 _step‚931 & lr-AdamW/pg1‚9.777258383198278e-06 & lr-AdamW/pg2‚9.777258383198278e-06£ò͸l4Ë-/ _wandbruntime‚13285  _runtime‚13285 b> (Ì-†P VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 214019775 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26088 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955591168 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚68193003 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1058716881793024 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13483552609280 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚639 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚934204189724160 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177047040 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1212 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 519365184 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12862274155736 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚947115488614912 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚947115488614912 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚934204189724160 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 234514448 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26088 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚637 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚934204171721216 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26090 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 519391274 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 401960960 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 % _timestamp‚1.7885236600686388e+09 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1187 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚926178457975044 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚68193640 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411420 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 234514446 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚932 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12911298890752 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 235151360 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007241 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13483317457920 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 145826133 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12863171567156 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚947114571204096 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 259091456 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1045233305243648 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 284876826 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚934204171721216 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407872 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 234514446 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12911298890752 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚947114571204096 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 _runtime‚13285.539134838 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407872 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1058716622701568 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955591168 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚913315269368528 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12910399482880 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450780 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0P”^¨ýÍ-‹P E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 145826135 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917410816 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410816 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 284876826 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚11349 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 284850738 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12910399482880 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚926177543524264 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214665 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1045233329183744 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 214019775 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26088 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955591168 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚68193003 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1058716881793024 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13483552609280 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚639 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚934204189724160 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177047040 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1212 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 519365184 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12862274155736 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚947115488614912 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚947115488614912 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚934204189724160 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 234514448 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26088 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚637 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 \HDeviceStatsMonitor.on_train_batch_star¥á4—t/allocated_bytes.large_pool.freed‚934204171721216 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26090 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 519391274 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 401960960 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 % _timestamp‚1.7885236600686388e+09 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1187 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚926178457975044 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚68193640 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411420 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 234514446 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚932 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12911298890752 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 235151360 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007241 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13483317457920 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 145826133 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12863171567156 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚947114571204096 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 259091456 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1045233305243648 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 284876826 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚934204171721216 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407872 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 234514446 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12911298890752 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚947114571204096 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 _runtime‚13285.539134838 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407872 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1058716622701568 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955591168 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚913315269368528 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12910399482880 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450780 JDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 284875329 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 284875329 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1058807111604736 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚947196231727104 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12912411707392 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12911512286208 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12911512286208 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚68199237 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 trainer/global_step‚11349 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13484487140352 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚934283820019712 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚934283802016768 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899421184 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚621 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 401960960 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 519435789 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12912411707392 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 234534344 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚947196231727104 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 519409673 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 284901443 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12864280228682 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007241 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26114 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 145838509 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1187 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1045322648404480 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26114 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚947195314302976 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 145838511 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12863382817054 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 _runtime‚13286.530109746 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955591168 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899421184 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917424128 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚926256632279710 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 234534344 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 214037748 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 214037125 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚68198616 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚934283820019712 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 519435789 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 234534346 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0åùl'Ð-”N K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 234534344 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 214037748 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 214037125 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚68198616 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚934283820019712 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 519435789 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 234534346 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 284875329 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 284875329 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1058807111604736 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚947196231727104 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12912411707392 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12911512286208 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12911512286208 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚68199237 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 trainer/global_step‚11349 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13484487140352 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚934283820019712 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚934283802016768 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899421184 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚621 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 401960960 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 519435789 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12912411707392 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 234534344 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚947196231727104 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 519409673 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 284901443 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12864280228682 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007241 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26114 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 145838509 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1187 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1045322648404480 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26114 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚947195314302976 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 145838511 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12863382817054 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 _runtime‚13286.530109746 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955591168 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899421184 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917424128 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚926256632279710 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564¥ê}34Ñ-/ _wandbruntime‚13286  _runtime‚13286WÕ¹Ú`Ò-Ú 0loss/FRt2/assignment_loss‚0.8045315646699497 0loss/FRt4/assignment_loss‚3.3884380138837376 1loss/SRqqt3/detection_loss‚1.5956876271300846 % _timestamp‚1.7885236610619297e+09 0loss/FBt2/detection_loss‚0.45032410393469036 /loss/FBt3/detection_loss‚2.0956757962703705 2loss/SRqqt3/assignment_loss‚1.1221028940131268 1loss/FBt2/assignment_loss‚0.04633638012193038 1loss/SRqqt4/detection_loss‚2.3267856339613595 /loss/FBt1/detection_loss‚3.6265241652727127 trainer/global_step‚11349 _runtime‚13286.532058635 /loss/FRt1/assignment_loss‚2.783636592626572 0loss/FRt3/assignment_loss‚2.5787299456803696 2loss/SRqqt2/assignment_loss‚0.7005974202358629 0loss/SRqqt1/detection_loss‚1.375818356126547 _step‚934 0loss/SRqqt2/detection_loss‚0.960443287777404 2loss/SRqqt1/assignment_loss‚0.9213263522088527 1loss/SRqqt4/assignment_loss‚1.706384385128816 3loss/FBt1/assignment_loss‚0.0023975424155651126 2loss/FBt4/assignment_loss‚0.001527572297277402 /loss/FRt2/detection_loss‚1.3840073389333807 .loss/FRt3/detection_loss‚2.246751344560281 /loss/FBt4/detection_loss‚3.4885890632867813 %loss/total_loss‚39.35885012270106 0loss/FBt3/assignment_loss‚0.7547486433031736 -loss/FRt1/detection_loss‚1.85281646980191 .loss/FRt4/detection_loss‚3.144669629060305 epoch‚9„ ZaeÓ-ß 0loss/FRt2/assignment_loss‚0.8045315646699497 0loss/FRt4/assignment_loss‚3.3884380138837376 1loss/SRqqt3/detection_loss‚1.5956876271300846 % _timestamp‚1.7885236610619297e+09 0loss/FBt2/detection_loss‚0.45032410393469036 /loss/FBt3/detection_loss‚2.0956757962703705 2loss/SRqqt3/assignment_loss‚1.1221028940131268 1loss/FBt2/assignment_loss‚0.04633638012193038 1loss/SRqqt4/detection_loss‚2.3267856339613595 /loss/FBt1/detection_loss‚3.6265241652727127 trainer/global_step‚11349 _runtime‚13286.532058635 /loss/FRt1/assignment_loss‚2.783636592626572 0loss/FRt3/assignment_loss‚2.5787299456803696 2loss/SRqqt2/assignment_loss‚0.7005974202358629 0loss/SRqqt1/detection_loss‚1.375818356126547 _step‚934 0loss/SRqqt2/detection_loss‚0.960443287777404 2loss/SRqqt1/assignment_loss‚0.9213263522088527 1loss/SRqqt4/assignment_loss‚1.706384385128816 3loss/FBt1/assignment_loss‚0.0023975424155651126 2loss/FBt4/assignment_loss‚0.001527572297277402 /loss/FRt2/detection_loss‚1.3840073389333807 .loss/FRt3/detection_loss‚2.246751344560281 /loss/FBt4/detection_loss‚3.4885890632867813 %loss/total_loss‚39.35885012270106 0loss/FBt3/assignment_loss‚0.7547486433031736 -loss/FRt1/detection_loss‚1.85281646980191 .loss/FRt4/detection_loss‚3.144669629060305 epoch‚9¦)6'¡Ô-‚(:’ éêÔÈã´ÿ gpu.0.gpu‚30.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚37.0 gpu.0.powerWatts‚117.81' gpu.0.enforcedPowerLimitWatts‚400.0) gpu.0.powerPercent‚29.452499999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0bq (hÕ-‚(:Ý éêÔŒ§í disk.rbd2.out‚ 3.87109375( proc.memory.availableMB‚ 1294.7734375# disk.nvme1n1.out‚67141.47265625 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.nvme1n1.in‚ 498.80078125 network.recv‚ 4.257768e+06 disk.dm-0.in‚ 1118.4375! disk.dm-0.out‚151928.44140625 disk.nvme0n1.out‚ 84786.96875 disk.sda2.in‚ 10.7578125 network.sent‚ 2.9889591e+07( disk./.usagePercent‚75.8810715654626" disk.sda2.out‚1520.00732421875" disk.sdb2.out‚1520.00732421875 disk.sda1.out‚0 disk.md0p1.out‚ 1465.3125! disk.sda.out‚1520.00732421875# proc.memory.rssMB‚ 2992.45703125 disk.md0.out‚ 1465.3125 disk.rbd2.in‚0 disk.sda1.in‚0 disk.nvme0n1.in‚ 619.63671875 disk.sdb.in‚ 7.0546875 disk.sda.in‚ 10.7578125 disk.rbd0.in‚0 disk.sdb2.in‚ 7.0546875 disk.md0.in‚17.8125! disk.sdb.out‚1520.00732421875$ disk./.usageGB‚2713.874668121338 disk.rbd0.out‚ 3.87109375$ memory_percent‚92.09733009338379 cpu‚25.21225292866196 proc.cpu.threads‚17) proc.memory.percent‚18.26450824737549 disk.md0p1.in‚17.8125÷³y6”Ö-‚(:‰ ŸéêÔ§ˆîÿ gpu.0.gpu‚47.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚159.419' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚39.85475 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0sh‰Žk×-‚(:à ŸéêÔœŠ¯€ disk.sdb2.in‚ 7.0546875! disk.sdb.out‚1521.49462890625 disk.rbd0.in‚0 disk.dm-0.in‚ 1118.4375 disk.md0p1.out‚ 1466.74609375 cpu‚25.212057494564327 disk.sda2.in‚ 10.7578125 disk.sdb1.out‚0 disk.md0.out‚ 1466.74609375 disk.rbd2.in‚0# proc.memory.rssMB‚ 2992.45703125 network.recv‚ 4.262707e+06) disk./.usagePercent‚75.85260162956814 disk.dm-0.out‚ 152143.328125 disk.md0p1.in‚17.8125 disk.sda.in‚ 10.7578125) proc.memory.percent‚18.26450824737549" disk./.usageGB‚2712.8564453125 disk.sda1.out‚0! disk.sda.out‚1521.49462890625 disk.nvme1n1.in‚ 498.80078125 disk.md0.in‚17.8125" disk.sdb2.out‚1521.49462890625 disk.nvme0n1.in‚ 619.63671875 disk.sdb.in‚ 7.0546875) proc.memory.availableMB‚ 1294.77734375 network.sent‚ 2.9984299e+07! disk.nvme0n1.out‚ 85001.703125" disk.sda2.out‚1521.49462890625 disk.rbd0.out‚ 3.87109375 disk.nvme1n1.out‚ 67141.625 disk.sdb1.in‚0$ memory_percent‚92.09730625152588 disk.sda1.in‚0 disk.rbd2.out‚ 3.87109375 proc.cpu.threads‚17n­gh”Ø-‚(:‰ ®éêÔŒÄËÿ gpu.0.gpu‚47.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚166.207' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚41.55175 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0—ÑQÙ-‚(:à ®éêÔéõŒ€$ disk./.usageGB‚2712.856513977051 disk.dm-0.in‚ 1118.4375 disk.rbd0.out‚ 3.87109375 disk.nvme1n1.in‚ 498.80078125#‚s¥‹Î proc.memory.rssMB‚ 2992.45703125 disk.sda1.in‚0 disk.nvme0n1.in‚ 619.63671875 disk.sda.in‚ 10.7578125# disk.nvme0n1.out‚85158.96484375 disk.sdb1.out‚0 disk.md0.out‚ 1468.34765625 disk.sdb2.in‚ 7.0546875) disk./.usagePercent‚75.85260354945773! disk.dm-0.out‚152300.74609375 disk.md0p1.in‚17.8125 disk.rbd2.in‚0 disk.rbd2.out‚ 3.87109375) proc.memory.availableMB‚ 1294.77734375 disk.md0p1.out‚ 1468.34765625 disk.sdb.out‚1523.158203125 disk.nvme1n1.out‚ 67141.78125 cpu‚25.211838934697216 proc.cpu.threads‚17 network.recv‚ 4.267598e+06 disk.sda.out‚1523.158203125 disk.sda2.in‚ 10.7578125$ memory_percent‚92.09730625152588) proc.memory.percent‚18.26450824737549 disk.sdb.in‚ 7.0546875 disk.sda2.out‚1523.158203125 disk.rbd0.in‚0 disk.sdb2.out‚1523.158203125 network.sent‚ 2.9990732e+07 disk.sda1.out‚0 disk.sdb1.in‚0 disk.md0.in‚17.8125JÀ9»’Ú-‚(:‡ ½éêÔ¯”ïÿ gpu.0.gpu‚51.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚171.168' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.792 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0TUVÛ-‚(:Ë ½éêÔÈò¨€" disk.nvme0n1.out‚ 85391.9921875 disk.sdb1.out‚0 disk.sdb2.in‚ 7.0546875 disk.sda1.in‚0 disk.sda1.out‚0) proc.memory.percent‚18.26450824737549 disk.sdb.in‚ 7.0546875 disk.sda.out‚ 1525.15625 disk.rbd2.out‚ 3.87109375 proc.cpu.threads‚17 network.sent‚ 2.9995994e+07 disk.nvme0n1.in‚ 619.63671875 disk.md0.out‚ 1470.28125 disk.rbd2.in‚0 network.recv‚ 4.270974e+06 disk.sdb.out‚ 1525.15625 disk.sda2.in‚ 10.7578125 disk.sda2.out‚ 1525.15625 disk.nvme1n1.in‚ 498.80078125 disk.sdb1.in‚0 disk.rbd0.in‚0 disk.dm-0.in‚ 1118.4375 disk.md0p1.in‚17.8125$ memory_percent‚92.09749698638916# proc.memory.rssMB‚ 2992.45703125% disk./.usageGB‚2712.7940101623535 disk.md0.in‚17.8125! disk.dm-0.out‚152529.76953125 disk.md0p1.out‚ 1470.28125 cpu‚25.21165634905018 disk.rbd0.out‚ 3.87109375 disk.sdb2.out‚ 1525.15625( disk./.usagePercent‚75.8508559166389 disk.sda.in‚ 10.7578125# disk.nvme1n1.out‚67142.02734375) proc.memory.availableMB‚ 1294.74609375/Æß®4Ü-/ _wandbruntime‚13343  _runtime‚13343}£â¿ÅÝ-¿ & lr-AdamW/pg1‚8.506539785754165e-06 & lr-AdamW/pg2‚8.506539785754165e-06 trainer/global_step‚11399 $ _timestamp‚1.788523718182552e+09 _runtime‚ 13343.653008 _step‚935" 'ÊÞ-Ä $ _timestamp‚1.788523718182552e+09 _runtime‚ 13343.653008 _step‚935 & lr-AdamW/pg1‚8.506539785754165e-06 & lr-AdamW/pg2‚8.506539785754165e-06 trainer/global_step‚11399§ÛÔí+4ß-/ _wandbruntime‚13343  _runtime‚13343J (à-†P F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 521592986 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 235509346 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12966953470976 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12917720812844 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 214923247 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1063240798958080 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1049699266280448 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12966953470976 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 235509348 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13541746857472 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 286083640 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 235509346 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚917214274074928 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214665 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26088 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚951152657972736 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚951151740561920 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 146444935 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 401960960 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411420 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚951152657972736 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚951151740561920 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚607 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 146444933 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚68478919 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177047040 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407872 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1063241037078016 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1049699290220544 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 % _timestamp‚1.7885237181837268e+09 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12966054063104 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410816 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 214923854 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚938185704501760 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26088 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955591168 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚938185686498816 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26090 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚938185686498816 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407872 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _runtime‚13343.654200105 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1212 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚917214291114288 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12966054063104 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚605 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171>,›f(á-‹P [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12966054063104 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚605 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 521592986 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 235509346 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12966953470976 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12917720812844 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 214923247 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1063240798958080 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1049699266280448 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12966953470976 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 235509348 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13541746857472 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 286083640 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 235509346 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚917214274074928 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214665 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26088 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚951152657972736 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚951151740561920 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 146444935 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 401960960 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411420 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚951152657972736 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚951151740561920 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚607 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 146444933 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚68478919 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177047040 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407872 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1063241037078016 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1049699290220544 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 % _timestamp‚1.7885237181837268e+09 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12966054063104 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410816 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 214923854 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚938185704501760 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26088 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955591168 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚938185686498816 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26090 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚938185686498816 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407872 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _runtime‚13343.654200105 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1212 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚917214291114288 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0¨­wQ4â-/ _wandbruntime‚13344  _runtime‚13344‚SÄĵ#ã-N OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917424128 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12968066287616 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899421184 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 235529246 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 286108231 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 235529244 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 286134345 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 235529246 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13542675624448 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚938265316794368 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚68483953 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 401960960 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411628 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 235529244 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚951233401084928 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177047040 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12967166866432 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1063331524398080 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚938265316794368 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 521637475 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955591168 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 286108231 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 146457311 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚68484581 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 286134345 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26114 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 214941892 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12919726885790 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12918829474162 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26114 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899421184 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177047040 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚917292254169056 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚628 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13542914956800 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 _step‚937 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12967166866432 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚951232483660800 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450988 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12968066287616 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 521663591 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007241 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 521663591 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚938265334797312 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚930211083643218 J@DeviceStatsMonitor.on_train_batch_end/allocation.sCöÈamall_pool.peak‚31686 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 214941262 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 % _timestamp‚1.7885237191850944e+09 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚951233401084928 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 146457309 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607rÊù0'ä-•N WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚938265316794368 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚68483953 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 401960960 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411628 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 235529244 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚951233401084928 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177047040 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12967166866432 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1063331524398080 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚938265316794368 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 521637475 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955591168 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 286108231 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 146457311 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚68484581 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 286134345 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26114 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 214941892 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12919726885790 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12918829474162 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26114 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899421184 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177047040 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚917292254169056 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚628 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13542914956800 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 _step‚937 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12967166866432 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚951232483660800 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450988 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12968066287616 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 521663591 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007241 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 521663591 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚938265334797312 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚930211083643218 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31686 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 214941262 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 % _timestamp‚1.7885237191850944e+09 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚951233401084928 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 146457309 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917424128 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12968066287616 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899421184 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 235529246 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 286108231 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 235529244 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 286134345 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 235529246 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13542675624448©—§DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚955188909919744 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚13401.527900865 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 287342630 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1187 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚942167219279360 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450780 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177047040 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955591168 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 523820788 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚955189827330560 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407872 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407872 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12974064881372 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚942167201276416 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13021708643328 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 215828064 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 287342630 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1054165251257344 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007241 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 215828675 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 % _timestamp‚1.7885237760572443e+09 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 523846878 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 236504246 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214665 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1212 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411420 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚942167201276416 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚921113278781328 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚955188909919744 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13022608051200 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 287316542 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 _step‚940 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚609 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 236504248 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410816 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917410816 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚68764331 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1067764968534016 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13599741216768 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955591168 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚955189827330560 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚611 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 401960960 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26088 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚955188909919744 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚13401.527900865 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 287342630 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1187 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚942167219279360 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450780 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177047040 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955591168 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 523820788 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚955189827330560 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407872 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407872 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12974064881372 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚942167201276416 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13021708643328 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 215828064 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 287342630 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1054165251257344 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007241 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 215828675 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 % _timestamp‚1.7885237760572443e+09 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 523846878 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 236504246 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214665 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1212 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411420 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚942167201276416 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚921113278781328 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚955188909919744 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13022608051200 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 287316542 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 _step‚940 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚609 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 236504248 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410816 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917410816 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚68764331 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1067764968534016 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13599741216768 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955591168 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚955189827330560 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚611 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 401960960 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728¬DÌŸŽ4ö-/ _wandbruntime‚13402  _runtime‚13402˵ð'÷-N H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 523891393 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚942246831571968 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1054254570478080 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚934166449457714 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450988 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 _runtime‚13402.485478777 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31686 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚68770658 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26114 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 _step‚941 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚955269653018624 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13600940065792 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 236524146 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13022821446656 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚942246831571968 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 523891393 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1187 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411628 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚605 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1067855486603776 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13601143746560 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 215846164 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 % _timestamp‚1.7885237770144446e+09 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚942246849574912 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177047040 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1054254546537984 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 203680768 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚921191258875456 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177047040 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 236524146 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚934165535006726 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 287341133 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899421184 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 287367247 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 523865277 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12974276131270 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007241 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚955270570442752 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚942246849574912 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13023720867840 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899421184 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26114 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955591168 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 147076111 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚921191275914816 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12975173542898 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13023720867840 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 236524144 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 147076109 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 227620864 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917424128 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171Ž¦%‡'ø-•N J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31686 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚68770658 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26114 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 _step‚941 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚955269653018624 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13600940065792 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 236524146 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13022821446656 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚942246831571968 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 523891393 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1187 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411628 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚605 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1067855486603776 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13601143746560 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 215846164 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 % _timestamp‚1.7885237770144446e+09 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚942246849574912 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177047040 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1054254546537984 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 203680768 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚921191258875456 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177047040 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 236524146 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚934165535006726 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 287341133 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899421184 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 287367247 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 523865277 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12974276131270 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007241 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚955270570442752 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚942246849574912 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13023720867840 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899421184 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26114 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955591168 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 147076111 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚921191275914816 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12975173542898 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13023720867840 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 236524144 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 147076109 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 227620864 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917424128 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 523891393 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚942246831571968 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1054254570478080 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚934166449457714 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450988 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 _runtime‚13402.485478777­£Ÿör4ù-/ _wandbruntime‚13402  _runtime‚13402¹X–Zú-Ô 2loss/SRqqt4/assignment_loss‚1.8153722723945975 /loss/FRt1/detection_loss‚1.8335573972789072 0loss/SRqqt3/detection_loss‚1.549165969213041 0loss/SRqqt4/detection_loss‚2.170231558382511 /loss/FBt3/detection_loss‚2.7769848704338074 2loss/SRqqt2/assignment_loss‚0.9165495345368981 .loss/FRt4/assignment_loss‚2.79893278837204 .loss/FRt2/detection_loss‚1.476974923077684 1loss/SRqqt1/detection_loss‚1.3350928127765656 trainer/global_step‚11449 1loss/SRqqt1/assignment_loss‚1.087112644507039 2loss/FBt4/assignment_loss‚0.038539463191834024 /loss/FBt1/detection_loss‚1.5794811579916213 -loss/FBt4/detection_loss‚3.46511971950531 &loss/total_loss‚36.968253517682356 _runtime‚13402.487017484 _step‚942 0loss/FRt1/assignment_loss‚2.1488628472600664 1loss/FBt2/assignment_loss‚0.04137826351821112 /loss/FRt2/assignment_loss‚1.518673566671518 1loss/FBt1/assignment_loss‚0.49796283157517407 1loss/FBt3/assignment_loss‚0.20900000460079582 % _timestamp‚1.7885237770169113e+09 /loss/FRt3/assignment_loss‚1.917473775645097 .loss/FRt4/detection_loss‚3.030322530269623 0loss/SRqqt2/detection_loss‚0.795107030096863 epoch‚9 2loss/SRqqt3/assignment_loss‚1.0002037689018135 .loss/FRt3/detection_loss‚2.270844408621391 /loss/FBt2/detection_loss‚0.6953093788599491DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007241 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917410816 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚959226079277568 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 288575532 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 288549444 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1212 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13077363223552 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚69048975 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 401960960 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 288575532 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955591168 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚634 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 _runtime‚13459.415077466 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 288549444 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 147682535 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411420 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1058631212294144 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 237499146 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177047040 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955591168 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0áªè9(‰.‹P WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 216730876 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450780 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚938040897614788 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚946148716054016 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407872 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚925012300527088 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚959226079277568 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 288575532 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 288549444 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1212 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13077363223552 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚69048975 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 401960960 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 288575532 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955591168 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚634 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 _runtime‚13459.415077466 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 288549444 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 147682535 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411420 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1058631212294144 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 237499146 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177047040 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955591168 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13657932426240 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13078262631424 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13078262631424 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13077363223552 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 526074680 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚925012283487728 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1058631188354048 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚959226996688384 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚946148734056960 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚632 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13029511538480 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214665 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407872 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚959226079277568 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26088 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚938041812065568 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 526048590 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 265382912 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13657690983424 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007241 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917410816 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 JDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899421184 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 526119195 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 237519046 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899421184 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26114 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13659081014272 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13658856361984 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚959306822376448 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31686 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955591168 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917424128 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13078476026880 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 288574035 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚620 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚946228364352512 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚946228364352512 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 237519044 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 147694911 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1058720531514880 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 _runtime‚13460.417298118 _step‚945 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 526119195 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13079375448064 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚946228346349568 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 216748998 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚925090280621216 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1187 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 % _timestamp‚1.7885238349461699e+09 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚959307739800576 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚69054709 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13029722788378 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177047040 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚622 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1072379612529152 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚938119986370234 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177047040 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 248592384 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 288600149 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 224652288 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚938120900821222 trainer/global_step‚11499 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 288600149 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1058720507574784 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 526093079 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚946228346349568 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 216749620 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1072379363936768 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26114 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13030620200006 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 147694909 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411628 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007241 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13079375448064 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 237519046 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 401960960 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728™¦îú'Œ.•N ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13079375448064 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 237519046 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 401960960 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 288574035 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899421184 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 526119195 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 237519046 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899421184 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26114 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13659081014272 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13658856361984 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚959306822376448 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31686 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955591168 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917424128 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13078476026880 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 288574035 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚620 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚946228364352512 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚946228364352512 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 237519044 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 147694911 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1058720531514880 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 _runtime‚13460.417298118 _step‚945 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 526119195 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13079375448064 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚946228346349568 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 216748998 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚925090280621216 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1187 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 % _timestamp‚1.7885238349461699e+09 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚959307739800576 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚69054709 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13029722788378 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177047040 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚622 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1072379612529152 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚938119986370234 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177047040 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 248592384 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 288600149 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 224652288 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚938120900821222 trainer/global_step‚11499 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 288600149 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1058720507574784 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 526093079 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚946228346349568 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 216749620 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1072379363936768 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26114 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13030620200006 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 147694909 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411628 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007241 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564±‰é`ü4./ _wandbruntime‚13460  _runtime‚13460&%R€]Ž.× 2loss/SRqqt2/assignment_loss‚0.9904438488936024 0loss/SRqqt3/detection_loss‚1.402399666151222 &loss/total_loss‚36.201775057497834 0loss/FRt1/assignment_loss‚2.5064362778383145 /loss/FRt4/assignment_loss‚2.526655069986979 0loss/FBt4/assignment_loss‚0.0931700225968623 .loss/FBt3/detection_loss‚2.125478581948714 _runtime‚13460.418876034 .loss/FRt3/detection_loss‚2.056526191186073 .loss/FBt4/detection_loss‚2.336850106716156 0loss/FRt2/assignment_loss‚1.0477199685573577 0loss/FRt3/assignment_loss‚1.4935581185096918 1loss/SRqqt4/assignment_loss‚1.716592556072606 .loss/FRt1/detection_loss‚2.086836889615584 _step‚946 % _timestamp‚1.7885238349487374e+09 2loss/SRqqt1/assignment_loss‚1.1742765071602272 0loss/SRqqt3/assignment_loss‚1.11250176185704 1loss/FBt1/assignment_loss‚0.26250687218804103 /loss/FRt4/detection_loss‚2.9529486974080403 0loss/SRqqt4/detection_loss‚2.781888465086619 1loss/SRqqt1/detection_loss‚1.5473362802128707 .loss/FBt1/detection_loss‚3.042401432991028 1loss/SRqqt2/detection_loss‚0.8897753053747895 epoch‚9 trainer/global_step‚11499 2loss/FBt2/assignment_loss‚0.046249159188806434 2loss/FBt3/assignment_loss‚0.013919180089136386 /loss/FRt2/detection_loss‚1.5208493886142969 0loss/FBt2/detection_loss‚0.47445470925377703ƒ§+¶b.Ü 0loss/FRt2/assignment_loss‚1.0477199685573577 0loss/FRt3/assignment_loss‚1.4935581185096918 1loss/SRqqt4/assignment_loss‚1.716592556072606 .loss/FRt1/detection_loss‚2.086836889615584 _step‚946 % _timestamp‚1.7885238349487374e+09 2loss/SRqqt1/assignment_loss‚1.1742765071602272 0loss/SRqqt3/assignment_loss‚1.11250176185704 1loss/FBt1/assignment_loss‚0.26250687218804103 /loss/FRt4/detection_loss‚2.9529486974080403 0loss/SRqqt4/detection_loss‚2.781888465086619 1loss/SRqqt1/detection_loss‚1.5473362802128707 .loss/FBt1/detection_loss‚3.042401432991028 1loss/SRqqt2/detection_loss‚0.8897753053747895 epoch‚9 trainer/global_step‚11499 2loss/FBt2/assignment_loss‚0.046249159188806434 2loss/FBt3/assignment_loss‚0.013919180089136386 /loss/FRt2/detection_loss‚1.5208493886142969 0loss/FBt2/detection_loss‚0.47445470925377703 2loss/SRqqt2/assignment_loss‚0.9904438488936024 0loss/SRqqt3/detection_loss‚1.402399666151222 &loss/total_loss‚36.201775057497834 0loss/FRt1/assignment_loss‚2.5064362778383145 /loss/FRt4/assignment_loss‚2.526655069986979 0loss/FBt4/assignment_loss‚0.0931700225968623 .loss/FBt3/detection_loss‚2.125478581948714 _runtime‚13460.418876034 .loss/FRt3/detection_loss‚2.056526191186073 .loss/FBt4/detection_loss‚2.336850106716156²{ {Ïq.‚(:æ ÄêêÔðŠüÿ# disk.nvme1n1.out‚68110.80078125) proc.memory.percent‚18.26450824737549 disk.rbd0.in‚0( disk./.usagePercent‚75.8526108023739 disk.sdb1.out‚0 disk.md0.out‚ 1484.46484375" disk.sdb2.out‚1539.90771484375 network.sent‚ 3.0251742e+07 disk.rbd2.out‚ 3.87109375 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.in‚ 10.7578125 disk.rbd0.out‚ 3.87109375$ memory_percent‚92.09978580474854# proc.memory.rssMB‚ 2992.45703125 disk.dm-0.out‚ 154792.0625 disk.md0p1.in‚17.8125 disk.md0p1.out‚ 1484.46484375# disk.nvme0n1.out‚86681.26171875! disk.sda.out‚1539.90771484375 network.recv‚ 4.310089e+06 disk.rbd2.in‚0 disk.dm-0.in‚ 1118.4375 disk.sdb.in‚ 7.0546875) proc.memory.availableMB‚ 1294.37109375 cpu‚25.209799833692532 proc.cpu.threads‚17! disk.sdb.out‚1539.90771484375 disk.md0.in‚17.8125 disk.sdb2.in‚ 7.0546875 disk.nvme0n1.in‚ 619.63671875 disk.nvme1n1.in‚ 498.80078125" disk.sda2.out‚1539.90771484375$ disk./.usageGB‚2712.856773376465 disk.sdb1.in‚0 disk.sda.in‚ 10.7578125Ä@ƒ’‘.‚(:‡ ÄêêÔ‰¹Ï€ gpu.0.gpu‚21.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚37.0 gpu.0.powerWatts‚114.408' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.602 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0åÕpÙ”’.‚(:‰ ÓêêÔá¦àÿ gpu.0.gpu‚44.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚170.559' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.63975 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ŽAþo“.‚(:ä ÓêêÔ›Á€ cpu‚25.20960172876027 network.recv‚ 4.31564e+06! disk.sdb2.out‚1541.8037109375 disk.sda1.out‚0 disk.rbd0.out‚ 3.87109375 disk.rbd2.in‚0 disk.nvme0n1.in‚ 619.63671875 disk.sdb.in‚ 7.0546875) proc.memory.availableMB‚ 1294.36328125# disk.nvme1n1.out‚68171.45703125! disk.nvme0n1.out‚ 86809.359375 proc.cpu.threads‚17 disk.dm-0.in‚ 1118.4375) disk./.usagePercent‚75.84867396213329! disk.sda2.out‚1541.8037109375 disk.rbd0.in‚0 disk.md0.in‚17.8125 disk.rbd2.out‚ 3.87109375 network.sent‚ 3.0361641e+07 disk.sda2.in‚ 10.7578125# proc.memory.rssMB‚ 2992.45703125) proc.memory.percent‚18.26450824737549% disk./.usageGB‚2712.7159729003906 disk.sda.out‚1541.8037109375 disk.sdb1.out‚0$ memory_percent‚92.09983348846436 disk.sda.in‚ 10.7578125 disk.nvme1n1.in‚ 498.80078125 disk.sdb2.in‚ 7.0546875! disk.dm-0.out‚154980.81640625 disk.md0p1.in‚17.8125 disk.md0p1.out‚ 1486.30078125 disk.sdb.out‚1541.8037109375 disk.sdb1.in‚0 disk.md0.out‚ 1486.30078125 disk.sda1.in‚0)dÐé’”.‚(:‡ âêêÔ‚ôý gpu.0.gpu‚16.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚173.344' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚43.336 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0æfÑas•.‚(:è âêêÔ¬Ïûþ network.recv‚ 4.320274e+06 disk.rbd2.out‚ 3.87109375# disk.nvme1n1.out‚68172.58203125 disk.sdb1.in‚0 disk.sda1.out‚0 disk.sdb.in‚ 7.0546875! disk.sdb.out‚1543.36181640625 disk.md0.out‚ 1487.79296875) disk./.usagePercent‚75.83500573492348! disk.sda.out‚1543.36181640625 proc.cpu.threads‚17 network.sent‚ 3.0365758e+07 disk.rbd2.in‚0" disk.sdb2.out‚1543.36181640625 disk.sda1.in‚0 disk.rbd0.in‚0 disk.md0p1.in‚17.8125 disk.sda.in‚ 10.7578125 disk.sdb1.out‚0 disk.md0.in‚17.8125 disk.sdb2.in‚ 7.0546875" disk.sda2.out‚1543.36181640625 disk.rbd0.out‚ 3.87109375# proc.memory.rssMB‚ 2992.45703125% disk./.usageGB‚2712.2271308898926 disk.md0p1.out‚ 1487.79296875' proc.memory.availableMB‚ 1294.109375) proc.memory.percent‚18.26450824737549 cpu‚25.20939380515206 disk.dm-0.in‚ 1118.4375! disk.dm-0.out‚155046.93359375 disk.nvme0n1.in‚ 619.63671875" disk.nvme0n1.out‚ 86894.3515625 disk.sda2.in‚ 10.7578125 disk.nvme1n1.in‚ 498.80078125$ memory_percent‚92.10138320922852n114–./ _wandbruntime‚13512  _runtime‚13512ÂœrÉ—.à _runtime‚13512.859231017 _step‚947 & lr-AdamW/pg1‚5.220945985800062e-06 & lr-AdamW/pg2‚5.220945985800062e-06 trainer/global_step‚11549 % _timestamp‚1.7885238873886743e+09H"‘Θ.È & lr-AdamW/pg1‚5.220945985800062e-06 & lr-AdamW/pg2‚5.220945985800062e-06 trainer/global_step‚11549 % _timestamp‚1.7885238873886743e+09 _runtime‚13512.859231017 _step‚947³‰bnÍ4™./ _wandbruntime‚13512  _runtime‚13512à (š.†P WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 224665600 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13084060784168 trainer/global_step‚11549 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚963264166046208 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1076812960029184 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410816 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13715810638336 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚941995348978296 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚941996263429076 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 289782346 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 217635777 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚69334444 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407872 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955591168 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450780 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26090 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 528276392 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚950130230831616 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177047040 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚950130230831616 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1063097173330944 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407872 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚619 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007241 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13716035303936 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚963263248635392 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13133017803776 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 148301335 _runtime‚13512.860119081 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13133917211648 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13133917211648 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1212 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚617 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0¿K„Úë›.‹P H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407872 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚619 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007241 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13716035303936 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚963263248635392 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13133017803776 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 148301335 _runtime‚13512.860119081 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13133917211648 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13133917211648 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1212 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚617 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 224665600 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13084060784168 trainer/global_step‚11549 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚963264166046208 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1076812960029184 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410816 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13715810638336 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚941995348978296 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚941996263429076 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 289782346 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 217635777 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚69334444 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407872 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955591168 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450780 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26090 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 528276392 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚950130230831616 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177047040 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚950130230831616 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1063097173330944 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚950209861127168 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 trainer/global_step‚11549 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177047040 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 289806937 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 _runtime‚13513.831810408 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 528320881 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899421184 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚942074437733742 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13134130607104 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899421184 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 256980992 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26114 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚928989285327616 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 148313711 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚626 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚69340081 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 401960960 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 217653790 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚69340707 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 528346997 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚963344909158400 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚950209879130112 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007241 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1076903689410560 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917424128 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955591168 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚950209879130112 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 528346997 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450988 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13085169445486 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚628 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1063186468611584 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚942075352184730 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 289806937 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚963343991734272 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 233040896 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13716963817984 _step‚949 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 289833051Û«'ž.”N EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚950209861127168 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 trainer/global_step‚11549 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177047040 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 289806937 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 _runtime‚13513.831810408 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 528320881 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899421184 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚942074437733742 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13134130607104 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899421184 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 256980992 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26114 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚928989285327616 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 148313711 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚626 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚69340081 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 401960960 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 217653790 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚69340707 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 528346997 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚963344909158400 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚950209879130112 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007241 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1076903689410560 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917424128 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955591168 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚950209879130112 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 528346997 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450988 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13085169445486 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚628 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1063186468611584 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚942075352184730 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 289806937 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚963343991734272 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 233040896 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13716963817984 _step‚949 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 289833051 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚950209861127168 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 238513946 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564µ(ò¦ô4Ÿ./ _wandbruntime‚13513  _runtime‚13513Äžñ2c .Ý epoch‚9 1loss/FBt1/assignment_loss‚0.11252485033440099 1loss/FBt4/assignment_loss‚0.14933554950590633 /loss/FRt1/detection_loss‚2.0049772041853657 /loss/FBt1/detection_loss‚1.7927777618169785 /loss/FBt3/detection_loss‚2.2536130036626543 % _timestamp‚1.7885238883626723e+09 /loss/FRt3/assignment_loss‚2.497005384308951 2loss/SRqqt1/assignment_loss‚1.3453889814205469 2loss/SRqqt2/assignment_loss‚0.7151596323298568 /loss/FRt3/detection_loss‚2.4941677898168564 1loss/SRqqt1/detection_loss‚1.7276940241456031 /loss/FBt4/detection_loss‚3.1397877410054207 1loss/SRqqt2/detection_loss‚0.8810706335161604 0loss/FBt2/detection_loss‚0.49677599526159927 2loss/SRqqt4/assignment_loss‚1.1651994560502077 &loss/total_loss‚36.809075893436976 0loss/FRt1/assignment_loss‚2.5234524774551392 2loss/SRqqt3/assignment_loss‚1.0975880335085093 /loss/FRt2/detection_loss‚1.3879648176287966 1loss/SRqqt4/detection_loss‚2.1996270920100964 trainer/global_step‚11549 _step‚950 0loss/FRt2/assignment_loss‚1.3333764278462954 0loss/FRt4/assignment_loss‚3.0217221776644387 0loss/SRqqt3/detection_loss‚1.549385352432728 _runtime‚13513.832872913 1loss/FBt2/assignment_loss‚0.14106304674433703 0loss/FBt3/assignment_loss‚0.1428578220405013 .loss/FRt4/detection_loss‚2.636560638745626JçÓÿh¡.â 2loss/SRqqt1/assignment_loss‚1.3453889814205469 2loss/SRqqt2/assignment_loss‚0.7151596323298568 /loss/FRt3/detection_loss‚2.4941677898168564 1loss/SRqqt1/detection_loss‚1.7276940241456031 /loss/FBt4/detection_loss‚3.1397877410054207 1loss/SRqqt2/detection_loss‚0.8810706335161604 0loss/FBt2/detection_loss‚0.49677599526159927 2loss/SRqqt4/assignment_loss‚1.1651994560502077 &loss/total_loss‚36.809075893436976 0loss/FRt1/assignment_loss‚2.5234524774551392 2loss/SRqqt3/assignment_loss‚1.0975880335085093 /loss/FRt2/detection_loss‚1.3879648176287966 1loss/SRqqt4/detection_loss‚2.1996270920100964 trainer/global_step‚11549 _step‚950 0loss/FRt2/assignment_loss‚1.3333764278462954 0loss/FRt4/assignment_loss‚3.0217221776644387 0loss/SRqqt3/detection_loss‚1.549385352432728 _runtime‚13513.832872913 1loss/FBt2/assignment_loss‚0.14106304674433703 0loss/FBt3/assignment_loss‚0.1428578220405013 .loss/FRt4/detection_loss‚2.636560638745626 epoch‚9 1loss/FBt1/assignment_loss‚0.11252485033440099 1loss/FBt4/assignment_loss‚0.14933554950590633 /loss/FRt1/detection_loss‚2.0049772041853657 /loss/FBt1/detection_loss‚1.7927777618169785 /loss/FBt3/detection_loss‚2.2536130036626543 % _timestamp‚1.7885238883626723e+09 /loss/FRt3/assignment_loss‚2.497005384308951¶Î+A3“¢.‚(:ˆ ñêêÔÆó〠gpu.0.gpu‚42.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚37.0 gpu.0.powerWatts‚115.966' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.9915 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÊqÕ0¦£.‚(:ß ñêêÔôèè disk.nvme1n1.in‚ 498.80078125# disk.nvme1n1.out‚68172.73046875$ memory_percent‚92.09823608398438) disk./.usagePercent‚75.83702417883337 disk.sda1.out‚0 disk.md0p1.in‚17.8125 disk.nvme0n1.in‚ 619.63671875 disk.sdb.in‚ 7.0546875 disk.sda.in‚ 10.7578125! disk.sda.out‚1544.83056640625 disk.rbd0.in‚0! disk.sdb.out‚1544.83056640625 disk.sdb1.out‚0 disk.md0.out‚ 1489.21875 disk.rbd2.in‚0" disk.sdb2.out‚1544.83056640625) proc.memory.percent‚18.26450824737549 cpu‚25.20916922116181" disk.sda2.out‚1544.83056640625 disk.rbd0.out‚ 3.87109375! disk.dm-0.out‚155142.31640žÄ625 disk.rbd2.out‚ 3.87109375 disk.sdb2.in‚ 7.0546875$ proc.memory.availableMB‚1294.625# proc.memory.rssMB‚ 2992.45703125 proc.cpu.threads‚17 disk.dm-0.in‚ 1118.4375" disk.nvme0n1.out‚ 86969.5859375 disk.sdb1.in‚0 disk.md0.in‚17.8125 disk.sda2.in‚ 10.7578125 network.sent‚ 3.0383876e+07 network.recv‚ 4.323439e+06% disk./.usageGB‚2712.2993202209473 disk.sda1.in‚0 disk.md0p1.out‚ 1489.21875´ ÒŽ”¤.‚(:‰ €ëêÔþòÁÿ gpu.0.gpu‚24.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚130.841' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚32.71025 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 Ú/Yt¥.‚(:é €ëêÔ‡Ç÷ÿ network.sent‚ 3.0478871e+07) disk./.usagePercent‚75.83541861784333 disk.rbd2.in‚0 disk.sda1.in‚0 disk.rbd0.out‚ 3.87109375 disk.nvme1n1.in‚ 498.80078125) proc.memory.availableMB‚ 1294.30078125" disk.sdb2.out‚1547.13525390625 disk.nvme0n1.in‚ 619.63671875! disk.sda.out‚1547.13525390625 proc.cpu.threads‚17 disk.dm-0.out‚155171.9609375 disk.md0p1.in‚17.8125# disk.nvme0n1.out‚86998.84765625 disk.sda.in‚ 10.7578125 disk.rbd0.in‚0 network.recv‚ 4.329176e+06# disk.nvme1n1.out‚68173.11328125 disk.dm-0.in‚ 1118.4375! disk.sdb.out‚1547.13525390625 disk.sdb1.out‚0$ disk./.usageGB‚2712.241897583008 disk.rbd2.out‚ 3.87109375 disk.sdb2.in‚ 7.0546875 disk.sda1.out‚0" disk.sda2.out‚1547.13525390625 disk.sdb1.in‚0$ memory_percent‚92.10021495819092# proc.memory.rssMB‚ 2992.45703125 disk.md0.in‚17.8125 disk.md0.out‚ 1491.45703125 disk.md0p1.out‚ 1491.45703125 disk.sdb.in‚ 7.0546875 disk.sda2.in‚ 10.7578125) proc.memory.percent‚18.26450824737549 cpu‚25.20892661424335~©)o“¦.‚(:ˆ ëêÔÈø¤ÿ gpu.0.gpu‚24.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚165.266' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚41.3165 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0§ æd§.‚(:Ù ëêÔ¶¿ïÿ# disk.nvme1n1.out‚68173.22265625 disk.sdb1.in‚0 disk.sdb2.in‚ 7.0546875 disk.rbd0.out‚ 3.87109375! disk.dm-0.out‚155295.69140625 disk.nvme1n1.in‚ 498.80078125 disk.sda2.out‚ 1548.62109375) proc.memory.availableMB‚ 1294.29296875 network.recv‚ 4.332976e+06 disk.nvme0n1.in‚ 619.63671875 disk.sdb.in‚ 7.0546875) proc.memory.percent‚18.26450824737549 disk.md0.in‚17.8125 disk.rbd2.in‚0 disk.sda1.in‚0 disk.sda.in‚ 10.7578125# proc.memory.rssMB‚ 2992.45703125) disk./.usagePercent‚75.84548747873615 disk.sdb2.out‚ 1548.62109375 disk.dm-0.in‚ 1118.4375 disk.md0p1.in‚17.8125 disk.md0p1.out‚ 1492.8828125 disk.sdb.out‚ 1548.62109375 disk.sda2.in‚ 10.7578125$ memory_percent‚92.10026264190674 disk.rbd2.out‚ 3.87109375 network.sent‚ 3.0484746e+07 disk.md0.out‚ 1492.8828125 disk.nvme0n1.out‚ 87122.46875 cpu‚25.208677518676154 proc.cpu.threads‚17 disk.rbd0.in‚0 disk.sdb1.out‚0 disk.sda1.out‚0 disk.sda.out‚ 1548.62109375# disk./.usageGB‚2712.60200881958^Ö^o”¨.‚(:‰ žëêÔ¯‹ºÿ gpu.0.gpu‚55.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚39.0 gpu.0.powerWatts‚170.559' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.63975 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ð÷ja©.‚(:Ö žëêÔ†îàÿ) disk./.usagePercent‚75.84553355608602$ disk./.usageGB‚2712.603656768799 disk.sda2.in‚ 10.7578125 disk.rbd0.out‚ 3.87109375) proc.memory.percent‚18.26450824737549 proc.cpu.threads‚17 disk.sdb1.out‚0 disk.sdb2.in‚ 7.0546875 disk.dm-0.in‚ 1118.4375# disk.nvme0n1.out‚87272.19140625 disk.sda2.out‚ 1550.0859375 network.recv‚ 4.338072e+06 disk.sdb1.in‚0 disk.dm-0.out‚155445.9296875 disk.nvme1n1.in‚ 498.80078125$ memory_percent‚92.10352897644043( proc.memory.availableMB‚ 1293.7578125 cpu‚25.20839233032669 disk.md0.in‚17.8125 disk.rbd2.in‚0 disk.nvme0n1.in‚ 619.63671875 disk.sdb.in‚ 7.0546875# proc.memory.rssMB‚ 2992.45703125 disk.sda1.in‚0 disk.md0p1.in‚17.8125 disk.sdb.out‚ 1550.0859375 disk.sda.in‚ 10.7578125 disk.sda.out‚ 1550.0859375 network.sent‚ 3.0491142e+07 disk.md0.out‚ 1494.2890625 disk.rbd2.out‚ 3.87109375 disk.sdb2.out‚ 1550.0859375 disk.md0p1.out‚ 1494.2890625 disk.rbd0.in‚0# disk.nvme1n1.out‚68173.73828125 disk.sda1.out‚0 É[«4ª./ _wandbruntime‚13565  _runtime‚13565׌~É«.à & lr-AdamW/pg1‚4.302029513919407e-06 & lr-AdamW/pg2‚4.302029513919407e-06 trainer/global_step‚11599 % _timestamp‚1.7885239400087857e+09 _runtime‚13565.480199413 _step‚951kóά.È % _timestamp‚1.7885239400087857e+09 _runtime‚13565.480199413 _step‚951 & lr-AdamW/pg1‚4.302029513919407e-06 & lr-AdamW/pg2‚4.302029513919407e-06 trainer/global_step‚11599·6p°4­./ _wandbruntime‚13565  _runtime‚13565£Ï1K (®.…P WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13140404852696 _runtime‚13565.48151859 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 291041336 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13188672384000 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 218542258 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚967300417993216 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚932810292900528 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007241 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13773946226688 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚954111763612160 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13139507441276 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 291015248 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 239488946 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 148920135 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1067563110427648 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚69622125 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1081337294774272 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚945950714792584 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450780 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚945949800341804 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407872 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1081337056654336 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1067563134367744 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1212 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 239488948 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955591168 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 214179840 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26090 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 % _timestamp‚1.7885239400104148e+09 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407872 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 239488948 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 291015248 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚69622734 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13188672384000 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚611 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 148920133 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214665 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚11599 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚954111745609216 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13189571791872 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚609 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 291041336 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 401960960 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917410816 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26088 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚954111763612160 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13774160406528 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 _step‚952 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26088 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410816 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚967301335404032 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚967300417993216 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚932810292900528 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007241 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13773946226688 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚954111763612160 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13139507441276 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 291015248 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 239488946 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 148920135 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1067563110427648 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚69622125 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1081337294774272 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚945950714792584 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450780 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚945949800341804 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407872 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1081337056654336 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1067563134367744 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1212 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 239488948 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955591168 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 214179840 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26090 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 % _timestamp‚1.7885239400104148e+09 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407872 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 239488948 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 291015248 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚69622734 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13188672384000 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚611 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 148920133 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214665 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚11599 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚954111745609216 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13189571791872 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚609 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 291041336 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 401960960 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917410816 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26088 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚954111763612160 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13774160406528 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 _step‚952 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26088 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410816 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚967301335404032 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚967382078516224 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚954191375904768 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 218560270 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚954191393907712 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚967381161092096 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚932888290034016 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 % _timestamp‚1.7885239409782567e+09 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 291065953 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚618 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1067652429648384 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 530548683 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚954191393907712 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917424128 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 233040896 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 530574799 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 256980992 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 401960960 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13190684608512 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13189785187328 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31686 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899421184 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 239508844 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007241 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 N>DeviceStatsMonitor.on_train_batchy|… _end/requested_bytes.all.peak‚ 13510214665 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚954191375904768 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 291039839 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚69627761 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13775094207488 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 trainer/global_step‚11599 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚616 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚946028889097250 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 218560888 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 239508846 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13190684608512 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1067652453588480 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 291039839 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 291065953 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1187 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899421184 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1081427780836864 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13140616102594 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13775327248384 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 239508846 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 530574799 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 148932509ËØÿ«'².•N WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚946029803548238 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13141513514222 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 148932511 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 530574799 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 256980992 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 401960960 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13190684608512 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13189785187328 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31686 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899421184 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 239508844 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007241 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚954191375904768 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 291039839 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚69627761 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13775094207488 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 trainer/global_step‚11599 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚616 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚946028889097250 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 218560888 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 239508846 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13190684608512 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1067652453588480 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 291039839 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 291065953 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1187 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899421184 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1081427780836864 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13140616102594 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13775327248384 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 239508846 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 530574799 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 148932509 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚967382078516224 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚954191375904768 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 218560270 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚954191393907712 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚967381161092096 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚932888290034016 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 % _timestamp‚1.7885239409782567e+09 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 291065953 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚618 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1067652429648384 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 530548683 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚954191393907712 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917424128 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 233040896 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1085861481902080 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917410816 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚958093260386816 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚971337587351040 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13244326964224 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 240483846 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1085861250073600 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13832178609152 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚958093260386816 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214665 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 292274238 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955591168 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚949905166156092 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚936709297606928 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 292248150 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 292248150 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 240483846 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 219447774 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚936709314646288 trainer/global_step‚11649 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177047040 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 _step‚956 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 532731996 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 240483848 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26088 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411420 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007241 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 240483848 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360>ÍkÛÁ.‹P QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 292274238 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955591168 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚949905166156092 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚936709297606928 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 292248150 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 292248150 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 Ag·¨6$7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 240483846 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 219447774 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚936709314646288 trainer/global_step‚11649 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177047040 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 _step‚956 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 532731996 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 240483848 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26088 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411420 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007241 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 240483848 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407872 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1212 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1072029095404544 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 % _timestamp‚1.7885239923811266e+09 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚958093278389760 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 401960960 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450780 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13195851509804 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚971338504761856 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1187 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1085861481902080 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917410816 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚958093260386816 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚971337587351040 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13244326964224 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 240483846 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1085861250073600 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13832178609152 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚958093260386816 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214665 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0¼Æ“z¸4Â./ _wandbruntime‚13618  _runtime‚13618¨’'Ã.N WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚971419247874048 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13833345171456 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚958172890682368 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚958172890682368 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007241 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 219466395 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚958172908685312 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26114 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955591168 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 532776485 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899421184 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13246339188736 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177047040 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 240503744 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 292272741 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 240503744 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31686 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1187 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411628 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13196062759702 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚610 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 _runtime‚13618.825396527 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚971418330449920 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚936787294740416 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1072118414625280 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 _step‚957 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13833567726592 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 532776485 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13245439767552 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13245439767552 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 532802601 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 219465785 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 401960960 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 trainer/global_step‚11649¶õÅ'Ä.•N =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26114 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 292272741 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1085951982351872 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 292298855 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 532776485 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13245439767552 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13245439767552 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 532802601 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 219465785 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 401960960 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 trainer/global_step‚11649 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚971419247874048 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13833345171456 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚958172890682368 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚958172890682368 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007241 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 219466395 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚958172908685312 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26114 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955591168 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 532776485 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899421184 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13246339188736 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177047040 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 240503744 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 292272741 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 240503744 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31686 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1187 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411628 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13196062759702 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚610 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 _runtime‚13618.825396527 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚971418330449920 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚936787294740416 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1072118414625280 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 _step‚957 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13833567726592 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 F5)_Æ.Ù 1loss/FBt1/assignment_loss‚0.20949049950907497 /loss/SRqqt4/detection_loss‚2.14214070908951 .loss/FRt3/detection_loss‚2.392268433339066 /loss/FRt4/detection_loss‚2.8315047209079447 trainer/global_step‚11649 /loss/FBt2/detection_loss‚0.5188353926704455 %loss/total_loss‚37.47111150020059 _step‚958 0loss/FRt3/assignment_loss‚2.6915494816170797 2loss/SRqqt3/assignment_loss‚1.3834042342579258 /loss/FRt2/detection_loss‚1.4696304387363948 .loss/FBt1/detection_loss‚2.312580934592656 .loss/FBt3/detection_loss‚2.699571192264557 1loss/SRqqt2/assignment_loss‚0.961362081579864 /loss/FRt1/detection_loss‚2.2112744599658365 1loss/SRqqt1/detection_loss‚1.4135028305378827 1loss/SRqqt2/detection_loss‚0.9197584962869276 1loss/SRqqt3/detection_loss‚1.5332440183098828 % _timestamp‚1.7885239933564005e+09 0loss/FRt4/assignment_loss‚2.5981747026626882 _runtime‚13618.826530722 0loss/FRt2/assignment_loss‚1.1279418039740177 1loss/SRqqt1/assignment_loss‚1.299055074400861 1loss/SRqqt4/assignment_loss‚1.567528795676701 1loss/FBt2/assignment_loss‚0.06918625002068149 2loss/FBt3/assignment_loss‚0.045146665484026016 3loss/FBt4/assignment_loss‚0.0016116806994302482 /loss/FBt4/detection_loss‚2.6532292791775296 /loss/FRt1/assignment_loss‚2.419119324439611 epoch‚9ÔAÑdÇ.Þ 0loss/FRt2/assignment_loss‚1.1279418039740177 1loss/SRqqt1/assignment_loss‚1.299055074400861 1loss/SRqqt4/assignment_loss‚1.567528795676701 1loss/FBt2/assignment_loss‚0.06918625002068149 2loss/FBt3/assignment_loss‚0.045146665484026016 3loss/FBt4/assignment_loss‚0.0016116806994302482 /loss/FBt4/detection_loss‚2.6532292791775296 /loss/FRt1/assignment_loss‚2.419119324439611 epoch‚9 1loss/FBt1/assignment_loss‚0.20949049950907497 /loss/SRqqt4/detection_loss‚2.14214070908951 .loss/FRt3/detection_loss‚2.392268433339066 /loss/FRt4/detection_loss‚2.8315047209079447 trainer/global_step‚11649 /loss/FBt2/detection_loss‚0.5188353926704455 %loss/total_loss‚37.47111150020059 _step‚958 0loss/FRt3/assignment_loss‚2.6915494816170797 2loss/SRqqt3/assignment_loss‚1.3834042342579258 /loss/FRt2/detection_loss‚1.4696304387363948 .loss/FBt1/detection_loss‚2.312580934592656 .loss/FBt3/detection_loss‚2.699571192264557 1loss/SRqqt2/assignment_loss‚0.961362081579864 /loss/FRt1/detection_loss‚2.2112744599658365 1loss/SRqqt1/detection_loss‚1.4135028305378827 1loss/SRqqt2/detection_loss‚0.9197584962869276 1loss/SRqqt3/detection_loss‚1.5332440183098828 % _timestamp‚1.7885239933564005e+09 0loss/FRt4/assignment_loss‚2.5981747026626882 _runtime‚13618.826530722¾ùBá”È.‚(:‰ ÚëêÔöÒÿ gpu.0.gpu‚54.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚37.0 gpu.0.powerWatts‚160.637' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚40.15925 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0c*¿8›É.‚(:ß ÚëêÔÄô¶€ disk.sda2.in‚ 10.7578125 disk.sda1.out‚0 disk.dm-0.in‚ 1118.4375 cpu‚25.20729018803098 disk.rbd0.in‚0 dCdþoÏisk.md0.out‚ 1500.86328125 disk.rbd2.out‚ 3.87109375$ memory_percent‚92.10376739501953 proc.cpu.threads‚17) disk./.usagePercent‚75.86584588116277 disk.sda.out‚1556.9189453125 disk.sdb1.out‚0) proc.memory.percent‚18.26450824737549 disk.sdb2.in‚ 7.0546875$ disk./.usageGB‚2713.330123901367 disk.nvme0n1.in‚ 619.63671875" disk.nvme0n1.out‚ 88128.1015625 disk.sdb.in‚ 7.0546875 disk.md0.in‚17.8125# proc.memory.rssMB‚ 2992.45703125 disk.rbd2.in‚0 disk.sda1.in‚0& proc.memory.availableMB‚ 1293.71875 network.recv‚ 4.356531e+06 disk.sdb.out‚1556.9189453125 disk.rbd0.out‚ 3.87109375 disk.md0p1.in‚17.8125! disk.sda2.out‚1556.9189453125! disk.nvme1n1.out‚ 68174.765625 disk.sdb1.in‚0 disk.dm-0.out‚156302.8671875! disk.sdb2.out‚1556.9189453125 network.sent‚ 3.0617012e+07 disk.md0p1.out‚ 1500.86328125 disk.nvme1n1.in‚ 498.80078125 disk.sda.in‚ 10.7578125낆’Ê.‚(:‡ éëêÔ÷˜†þ gpu.0.gpu‚53.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚171.168' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.792 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0,ÆxÙrË.‚(:ç éëêÔ”‘ÿ disk.sda1.in‚0 disk.dm-0.in‚ 1118.4375 disk.sdb1.out‚0 disk.rbd2.in‚0# proc.memory.rssMB‚ 2992.45703125 network.sent‚ 3.0724194e+07) disk./.usagePercent‚75.86453939630479 disk.md0p1.out‚ 1502.3359375 disk.sda2.in‚ 10.7578125# disk.nvme1n1.out‚68174.92578125 disk.md0.out‚ 1502.3359375% disk./.usageGB‚2713.2833976745605 disk.sda.in‚ 10.7578125" disk.sda2.out‚1558.44384765625 disk.nvme1n1.in‚ 498.80078125 cpu‚25.20704928086858 proc.cpu.threads‚17 disk.md0p1.in‚17.8125 disk.nvme0n1.in‚ 619.63671875 network.recv‚ 4.36124e+06 disk.sda1.out‚0 disk.rbd0.out‚ 3.87109375 disk.sdb1.in‚0) proc.memory.availableMB‚ 1293.44140625" disk.nvme0n1.out‚ 88285.9140625 disk.md0.in‚17.8125$ memory_percent‚92.10546016693115! disk.dm-0.out‚156460.83984375! disk.sda.out‚1558.44384765625 disk.sdb2.in‚ 7.0546875" disk.sdb2.out‚1558.44384765625) proc.memory.percent‚18.26450824737549 disk.sdb.in‚ 7.0546875! disk.sdb.out‚1558.44384765625 disk.rbd0.in‚0 disk.rbd2.out‚ 3.87109375“pO’Ì.‚(:‡ øëêÔé¯þ gpu.0.gpu‚2.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚36.0 gpu.0.powerWatts‚113.798' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.4495 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0÷bºhÍ.‚(:Ý øëêԣ͑ÿ! disk.nvme1n1.out‚ 68247.015625 disk.sdb1.out‚0 disk.rbd2.out‚ 3.87109375 disk.sda1.in‚0 network.recv‚ 4.365598e+06# disk.nvme0n1.out‚88414.31640625 disk.sda.out‚ 1560.45703125# memory_percent‚92.1039342880249) proc.memory.availableMB‚ 1293.69140625# proc.memory.rssMB‚ 2992.45703125 disk.sdb1.in‚0 disk.md0.out‚ 1504.29296875 disk.rbd2.in‚0 disk.sdb.out‚ 1560.45703125 network.sent‚ 3.0730244e+07 disk.md0p1.in‚17.8125 disk.nvme0n1.in‚ 619.63671875 disk.sdb.in‚ 7.0546875 disk.sda2.in‚ 10.7578125 disk.sdb2.out‚ 1560.45703125 proc.cpu.threads‚17% disk./.usageGB‚2713.3534088134766! disk.dm-0.out‚156661.33203125 disk.sda.in‚ 10.7578125 disk.rbd0.in‚0 disk.sdb2.in‚ 7.0546875 disk.sda2.out‚ 1560.45703125 disk.md0.in‚17.8125 disk.sda1.out‚0) proc.memory.percent‚18.26450824737549 cpu‚25.206822169609026 disk.rbd0.out‚ 3.87109375) disk./.usagePercent‚75.86649693705083 disk.dm-0.in‚ 1118.4375 disk.md0p1.out‚ 1504.29296875 disk.nvme1n1.in‚ 498.80078125j‘襒Î.‚(:‡ ‡ìêÔùÆþ gpu.0.gpu‚23.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚37.0 gpu.0.powerWatts‚123.712' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚30.928 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ZéIøqÏ.‚(:æ ‡ìêÔú’Œÿ disk.dm-0.in‚ 1118.4375 disk.md0p1.out‚ 1505.76171875" disk.nvme0n1.out‚ 88600.7265625 disk.sdb2.in‚ 7.0546875 cpu‚25.206595734034057 proc.cpu.threads‚17" disk.sdb2.out‚1561.98193359375) disk./.usagePercent‚75.86830579301508 disk.md0.out‚ 1505.76171875 disk.sda1.in‚0) proc.memory.availableMB‚ 1293.69140625 disk.md0p1.in‚17.8125 disk.sdb.in‚ 7.0546875 disk.rbd0.in‚0 disk.md0.in‚17.8125! disk.sdb.out‚1561.98193359375" disk.nvme1n1.out‚ 68275.1171875 disk.sda1.out‚0# proc.memory.rssMB‚ 2992.45703125 network.sent‚ 3.0736124e+07 network.recv‚ 4.369638e+06 disk.sda.in‚ 10.7578125 disk.rbd2.out‚ 3.87109375! disk.sda.out‚1561.98193359375" disk.sda2.out‚1561.98193359375 disk.rbd0.out‚ 3.87109375 disk.nvme1n1.in‚ 498.80078125 disk.sdb1.out‚0 disk.rbd2.in‚0) proc.memory.percent‚18.26450824737549 disk.dm-0.out‚ 156875.84375 disk.nvme0n1.in‚ 619.63671875 disk.sda2.in‚ 10.7578125 disk.sdb1.in‚0% disk./.usageGB‚2713.4181022644043# memory_percent‚92.1039342880249ãƒ4Ð./ _wandbruntime‚13677  _runtime‚13677×”uSÈÑ. & lr-AdamW/pg1‚2.729795302719925e-06 & lr-AdamW/pg2‚2.729795302719925e-06 trainer/global_step‚11699 $ _timestamp‚1.788524052323013e+09 _runtime‚13677.793531869 _step‚959¼êº!ÍÒ.Ç $ _timestamp‚1.788524052323013e+09 _runtime‚13677.793531869 _step‚959 & lr-AdamW/pg1‚2.729795302719925e-06 & lr-AdamW/pg2‚2.729795302719925e-06 trainer/global_step‚11699¿±Ÿ4Ó./ _wandbruntime‚13677  _runtime‚13677b´¦ (Ô.†P E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1076495056441344 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚975375674119680 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955591168 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 150157733 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1212 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 _runtime‚13677.799009734 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚962074793167360 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚962074775164416 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13300880952320 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1076495032501248 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407872 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 241478746 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚940608302313328 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13250400755492 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13300880952320 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1090385431174144 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407872 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚975374756708864 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26088 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚953859617519600 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450780 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955591168 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13299981544448 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 401960960 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚70195420 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410816 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚962074793167360 _step‚960 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚975374756708864 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 % _timestamp‚1.7885240523244781e+09 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 241478746 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13890625435648 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26088 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177047040 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 220353153 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13251298166912 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 293507140 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚953858703068820 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 trainer/global_step‚11699 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 534959798 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚975375674119680 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 293481052 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚962074775164416 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13299981544448 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1090385681876992 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 226762752 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917410816 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚612 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411420 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007241 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 293481052 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚975374756708864 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 % _timestamp‚1.7885240523244781e+09 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 241478746 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13890625435648 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26088 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177047040 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 220353153 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13251298166912 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 293507140 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚953858703068820 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 trainer/global_step‚11699 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 534959798 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚975375674119680 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 293481052 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚962074775164416 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13299981544448 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1090385681876992 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 226762752 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917410816 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚612 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411420 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007241 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 293481052 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090Àýáÿ4Ö./ _wandbruntime‚13679  _runtime‚13679«ó¾y×.N ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚962154405459968 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177047040 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 150170111 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚975456417231872 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13251509416810 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚641 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚639 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955591168 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.largBÈ”öe_pool.peak‚ 13037993984 trainer/global_step‚11699 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 220371242 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 _step‚961 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 293531757 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 293505643 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚962154405459968 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177047040 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 535030403 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1076584375662080 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 228846592 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 EW'Ø.•N RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177047040 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 150170111 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚975456417231872 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13251509416810 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚641 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚639 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955591168 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 trainer/global_step‚11699 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 220371242 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 _step‚961 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 293531757 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 293505643 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚962154405459968 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177047040 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 535030403 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1076584375662080 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 228846592 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0ÁMˆ4Ù./ _wandbruntime‚13679  _runtime‚13679Tþ‹r\Ú.Ö 1loss/FBt1/assignment_loss‚0.13841179988287067 _runtime‚13679.106747433 /loss/FRt2/detection_loss‚1.4923471641359907 %loss/total_loss‚37.69527621795605 _step‚962 2loss/SRqqt2/assignment_loss‚0.7102535933015808 1loss/FBt3/assignment_loss‚0.08501759307459149 /loss/FRt2/assignment_loss‚1.269983565265482 2loss/SRqqt1/assignment_loss‚1.1285675676795197 1loss/SRqqt1/detection_loss‚1.4679734768966834 1loss/SRqqt3/detection_loss‚1.8343587674848412 0loss/SRqqt4/detection_loss‚2.496619738638401 1loss/SRqqt4/assignment_loss‚1.210176650279512 /loss/FRt3/detection_loss‚2.2707247398793697 /loss/FRt4/detection_loss‚3.1196796844402948 /loss/FBt2/detection_loss‚0.6250710707711244 .loss/FBt3/detection_loss‚2.197779791222678 .loss/FBt4/detection_loss‚3.224413442611694 /loss/FRt4/assignment_loss‚2.805254499117533 % _timestamp‚1.7885240536363645e+09 0loss/FBt4/assignment_loss‚0.8877026238309554 epoch‚9 trainer/global_step‚11699 0loss/FRt1/assignment_loss‚2.2153393431143327 2loss/SRqqt3/assignment_loss‚1.0961486834060887 0loss/FBt2/assignment_loss‚0.1517448294637103 /loss/FRt1/detection_loss‚1.7996337201487966 1loss/SRqqt2/detection_loss‚0.9705235819903112 .loss/FBt1/detection_loss‚2.207202099263668 0loss/FRt3/assignment_loss‚2.2903481920560202¤çŽaÛ.Û 1loss/SRqqt4/assignment_loss‚1.210176650279512 /loss/FRt3/detection_loss‚2.2707247398793697 /loss/FRt4/detection_loss‚3.1196796844402948 /loss/FBt2/detection_loss‚0.6250710707711244 .loss/FBt3/detection_loss‚2.197779791222678 .loss/FBt4/detection_loss‚3.224413442611694 /loss/FRt4/assignment_loss‚2.805254499117533 % _timestamp‚1.7885240536363645e+09 0loss/FBt4/assignment_loss‚0.8877026238309554 epoch‚9 trainer/global_step‚11699 0loss/FRt1/assignment_loss‚2.2153393431143327 2loss/SRqqt3/assignment_loss‚1.0961486834060887 0loss/FBt2/assignment_loss‚0.1517448294637103 /loss/FRt1/detection_loss‚1.7996337201487966 1loss/SRqqt2/detection_loss‚0.9705235819903112 .loss/FBt1/detection_loss‚2.207202099263668 0loss/FRt3/assignment_loss‚2.2903481920560202 1loss/FBt1/assignment_loss‚0.13841179988287067 _runtime‚13679.106747433 /loss/FRt2/detection_loss‚1.4923471641359907 %loss/total_loss‚37.69527621795605 _step‚962 2loss/SRqqt2/assignment_loss‚0.7102535933015808 1loss/FBt3/assignment_loss‚0.08501759307459149 /loss/FRt2/assignment_loss‚1.269983565265482 2loss/SRqqt1/assignment_loss‚1.1285675676795197 1loss/SRqqt1/detection_loss‚1.4679734768966834 1loss/SRqqt3/detection_loss‚1.8343587674848412 0loss/SRqqt4/detection_loss‚2.496619738638401Â-ÍN×”Ü.‚(:‰ –ìêÔ©Çöÿ gpu.0.gpu‚54.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚156.911' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚39.22775 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‡ N—lÝ.‚(:á –ìêÔ†‡ù disk.dm-0.in‚ 1118.4375 disk.nvme0n1.in‚ 619.63671875 disk.sdb.in‚ 7.0546875 disk.sda2.in‚ 10.7578125) disk./.usagePercent‚75.87356042413006 disk.md0.in‚17.8125 disk.rbd2.out‚ 3.87109375 disk.sdb2.in‚ 7.0546875 disk.sda.in‚ 10.7578125 disk.rbd0.out‚ 3.87109375' proc.memory.availableMB‚ 1293.203125 proc.cpu.threads‚17 network.recv‚ 4.373701e+06 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.sdb2.out‚1563.490234375 disk.sda1.out‚0$ memory_percent‚92.10691452026367 disk.rbd2.in‚0 disk.sda1.in‚0 disk.dm-0.out‚157111.4140625 disk.md0p1.in‚17.8125 cpu‚25.206395172071854 network.sent‚ 3.0755076e+07 disk.md0p1.out‚ 1507.21484375# disk.nvme1n1.out‚68352.98046875# proc.memory.rssMB‚ 2992.45703125) proc.memory.percent‚18.26450824737549 disk.sda.out‚1563.490234375 disk.sda2.out‚1563.490234375 disk.rbd0.in‚0% disk./.usageGB‚2713.6060333251953# disk.nvme0n1.out‚88758.43359375 disk.sdb.out‚1563.490234375 disk.md0.out‚ 1507.21484375 disk.nvme1n1.in‚ 498.80078125c†|5”Þ.‚(:‰ ¥ìêÔ‹®Åÿ gpu.0.gpu‚33.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚37.0 gpu.0.powerWatts‚115.025' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.75625 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¹Ó8à\ß.‚(:Ñ ¥ìêÔÔñð€ disk.sdb1.in‚0 disk.sda.out‚ 1565.609375 disk.rbd0.in‚0 disk.sdb.out‚ 1565.609375 cpu‚25.20619182979894 disk.nvme1n1.in‚ 498.80078125# disk.nvme0n1.out‚88968.18359375# disk.nvme1n1.out‚68404.82421875 disk.rbd0.out‚ 3.87109375 disk.md0.in‚17.8125 disk.md0p1.in‚17.8125 disk.md0p1.out‚ 1509.2734375# memory_percent‚92.1025276184082 disk.rbd2.out‚ 3.87109375 disk.sdb2.out‚ 1565.609375 disk.sda1.in‚0 disk.sda1.out‚0 disk.nvme0n1.in‚ 619.63671875# proc.memory.rssMB‚ 2992.45703125) proc.memory.percent‚18.26450824737549 network.recv‚ 4.378132e+06% disk./.usageGB‚2713.6054611206055 disk.sda.in‚ 10.7578125 disk.rbd2.in‚0 disk.sdb.in‚ 7.0546875' proc.memory.availableMB‚ 1293.921875 proc.cpu.threads‚17 network.sent‚ 3.0850672e+07 disk.sda2.in‚ 10.7578125 disk.sdb1.out‚0 disk.dm-0.in‚ 1118.4375) disk./.usagePercent‚75.87354442505024 disk.sda2.out‚ 1565.609375 disk.dm-0.out‚157373.0078125 disk.md0.out‚ 1509.2734375 disk.sdb2.in‚ 7.0546875.ßsà.‚(:è ´ìêԩȶ€ disk.sdb1.in‚0 disk.rbd2.in‚0# proc.memory.rssMB‚ 2992.45703125 proc.cpu.threads‚17 disk.nvme1n1.in‚ 498.80078125" disk.nvme1n1.out‚ 68534.8359375 disk.dm-0.in‚ 1118.4375! disk.sdb.out‚1567.18408203125 disk.sda2.in‚ 10.7578125 cpu‚25.205986611694666 disk.sdb2.in‚ 7.0546875 disk.md0.in‚17.8125! disk.dm-0.out‚157660.33984375 disk.nvme0n1.in‚ 619.63671875" disk.sda2.out‚1567.18408203125 disk.sdb.in‚ 7.0546875 disk.sda.in‚ 10.7578125$ memory_percent‚92.10114479064941) proc.memory.percent‚18.26450824737549 network.sent‚ 3.0854506e+07 network.recv‚ 4.382807e+06) disk./.usagePercent‚75.88056300804544% disk./.usageGB‚2713.8564796447754 disk.sdb1.out‚0 disk.md0.out‚ 1510.7890625 disk.sda1.in‚0 disk.md0p1.in‚17.8125" disk.sdb2.out‚1567.18408203125 disk.md0p1.out‚ 1510.7890625# disk.nvme0n1.out‚89125.50390625! disk.sda.out‚1567.18408203125 disk.rbd0.in‚0 disk.rbd0.out‚ 3.87109375( proc.memory.availableMB‚ 1294.1484375 disk.rbd2.out‚ 3.87109375 disk.sda1.out‚0‘›ˆ…“á.‚(:ˆ ´ìêÔӀ݊ gpu.0.gpu‚2.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚37.0 gpu.0.powerWatts‚163.597' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚40.89925 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Dõ’â.‚(:‡ ÃìêÔàƒâÿ gpu.0.gpu‚30.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚163.09' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚40.7725 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚00¯öïlã.‚(:á ÃìêÔ‚±ã€# proc.memory.rssMB‚ 2992.45703125 disk.rbd0.in‚0 disk.md0.in‚17.8125 proc.cpu.threads‚17 disk.sdb.out‚1568.7197265625 disk.sdb.in‚ 7.0546875! disk.sda2.out‚1568.7197265625# disk.nvme1n1.out‚68663.66796875 disk.sdb2.in‚ 7.0546875) proc.memory.percent‚18.26450824737549 cpu‚25.205742375008708 network.sent‚ 3.086041e+07% disk./.usageGB‚2713.8554649353027 disk.sdb1.in‚0 disk.rbd2.out‚ 3.87109375) proc.memory.availableMB‚ 1294.15234375 disk.rbd0.out‚ 3.87109375 disk.nvme0n1.in‚ 619.63671875! disk.nvme0n1.out‚ 89221.515625 network.recv‚ 4.386769e+06 disk.sda2.in‚ 10.7578125 disk.sdb1.out‚0 disk.rbd2.in‚0 disk.dm-0.in‚ 1118.4375! disk.dm-0.out‚157885.18359375 disk.md0p1.out‚ 1512.2734375 disk.sda.in‚ 10.7578125 disk.sda.out‚1568.7197265625 disk.nvme1n1.in‚ 498.80078125! disk.sdb2.out‚1568.7197265625 disk.sda1.in‚0 disk.sda1.out‚0# memory_percent‚92.1011209487915 disk.md0p1.in‚17.8125( disk./.usagePercent‚75.8805346363439 disk.md0.out‚ 1512.2734375Ç ú4ä./ _wandbruntime‚13737  _runtime‚13737eLÍ^Êå.Ä ' lr-AdamW/pg1‚2.0768052146542248e-06 ' lr-AdamW/pg2‚2.0768052146542248e-06 trainer/global_step‚11749 $ _timestamp‚1.788524111590921e+09 _runtime‚13737.061321654 _step‚9636dóÏæ.É ' lr-AdamW/pg2‚2.0768052146542248e-06 trainer/global_step‚11749 $ _timestamp‚1.788524111590921e+09 _runtime‚13737.061321654 _step‚963 ' lr-AdamW/pg1‚2.0768052146542248e-06Ã3È4ç./ _wandbruntime‚13737  _runtime‚13737 ¯®ßä#è.„P TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 $ _timestamp‚1.788524111592036e+09 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚979411926066688 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 242473648 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26088 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 242473646 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚966056289942016 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13306744824020 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚979412843477504 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 150776533 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13305847412600 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚966056307944960 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚979412843477504 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410816 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13356535532544 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1094909678953984 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 401960960 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26088 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955591168 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13355636124672 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955591168 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 242473646 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚70483291 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 256994304 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚629 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 294740042 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26090 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 294740042 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚966056289942016 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚979411926066688 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1187 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚957814068883108 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 294713954 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407872 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚944507307019728 trainer/global_step‚11749 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917410816 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 221259824 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1212 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _step‚964 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 221260453 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13948428421632 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13t&355636124672 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1080961017478144 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 _runtime‚13737.06245775 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1094909421959680 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 537213690 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 150776535 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚957813154432328 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚944507324059088 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450780Ìå]¬(é.‰P O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚979411926066688 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 242473648 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26088 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 242473646 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚966056289942016 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13306744824020 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚979412843477504 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 150776533 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13305847412600 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚966056307944960 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚979412843477504 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410816 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13356535532544 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1094909678953984 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 401960960 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26088 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955591168 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13355636124672 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955591168 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 242473646 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚70483291 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 256994304 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚629 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 294740042 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26090 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 294740042 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚966056289942016 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚979411926066688 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1187 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚957814068883108 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 294713954 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407872 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚944507307019728 trainer/global_step‚11749 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917410816 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 221259824 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1212 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _step‚964 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 221260453 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13948428421632 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13355636124672 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1080961017478144 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 _runtime‚13737.06245775 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1094909421959680 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 537213690 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 150776535 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚957813154432328 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚944507324059088 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450780 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 $ _timestamp‚1.788524111592036e+09Ä 8ò4ê./ _wandbruntime‚13738  _runtime‚13738xíŠX'ë.N dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1081050336698880 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚966135920237568 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177047040 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 537232089 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 401960960 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 _runtime‚13738.169352336 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1094999892492288 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899421184 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚966135938240512 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 150788911 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 150788909 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 254883840 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 537258205 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 221278592 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 537258205 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 294764659 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 221277955 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13306956073918 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚979492669165568 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚957893157638762 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26114 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13356748928000 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 242493546 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚635 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚944585304153216 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13949579733504 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450988 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177047040 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 294764659 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 242493544 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 294738545 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚637 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚70489681 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1081050312758784 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚979493586589696 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 242493546 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007241 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚966135938240512 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13357648349184 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 294738545 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1187 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 537232089 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13357648349184 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31686 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚70489046 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚944585287113856 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 trainer/global_step‚11749 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚957892243187774 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13307853485546 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26114 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13356748928000 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 _step‚965 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0‰Ñ@7'ì.•N N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899421184 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚966135938240512 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 150788911 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 150788909 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 254883840 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 537258205 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 221278592 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 537258205 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 294764659 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 221277955 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13306956073918 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚979492669165568 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚957893157638762 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26114 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13356748928000 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 242493546 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚635 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚944585304153216 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13949579733504 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450988 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177047040 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 294764659 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 242493544 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 294738545 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚637 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚70489681 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1081050312758784 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚979493586589696 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 242493546 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007241 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚966135938240512 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13357648349184 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 294738545 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1187 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 537232089 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13357648349184 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31686 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚70489046 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚944585287113856 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 trainer/global_step‚11749 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚957892243187774 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13307853485546 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26114 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13356748928000 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 _step‚965 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1081050336698880 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚966135920237568 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177047040 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 537232089 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 401960960 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 _runtime‚13738.169352336 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1094999892492288 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171Å6è,é4í./ _wandbruntime‚13738  _runtime‚137385U–î.Ý 0loss/FRt2/assignment_loss‚1.1780830698135571 0loss/FRt4/assignment_loss‚2.2661130568560433 .loss/FRt1/detection_loss‚1.748107379540615 .loss/FRt4/detection_loss‚3.435303014867446 _step‚966 1loss/SRqqt4/detection_loss‚2.8059141508170535 epoch‚9 % _timestamp‚1.7885241127013347e+09 2loss/SRqqt1/assignment_loss‚0.8817929527722299 2loss/SRqqt2/assignment_loss‚0.7935960717956748 2loss/SRqqt3/assignment_loss‚0.9676659725916882 /loss/FRt3/detection_loss‚2.2530659404159947 /loss/FBt2/detection_loss‚0.5100896450678339 1loss/FBt2/assignment_loss‚0.06896007431835673 1loss/FBt4/assignment_loss‚0.07228031336890126 /loss/FRt2/detection_loss‚2.1113207267167478 1loss/SRqqt1/detection_loss‚1.7648715823042123 _runtime‚13738.171455746 0loss/FRt1/assignment_loss‚2.4042266742749647 /loss/FRt3/assignment_loss‚2.035198963501237 2loss/FBt3/assignment_loss‚0.038439213199519776 1loss/SRqqt2/detection_loss‚1.0002771943415467 1loss/SRqqt3/detection_loss‚1.7435310398538908 /loss/FBt1/detection_loss‚2.5635280476676092 /loss/FBt3/detection_loss‚2.0490279472791233 trainer/global_step‚11749 2loss/SRqqt4/assignment_loss‚1.3736586460754985 1loss/FBt1/assignment_loss‚0.1110731TàÊb1846957499 /loss/FBt4/detection_loss‚2.3282621542612714 %loss/total_loss‚36.50438695017059¸=ê&hï.â 1loss/SRqqt2/detection_loss‚1.0002771943415467 1loss/SRqqt3/detection_loss‚1.7435310398538908 /loss/FBt1/detection_loss‚2.5635280476676092 /loss/FBt3/detection_loss‚2.0490279472791233 trainer/global_step‚11749 2loss/SRqqt4/assignment_loss‚1.3736586460754985 1loss/FBt1/assignment_loss‚0.11107311846957499 /loss/FBt4/detection_loss‚2.3282621542612714 %loss/total_loss‚36.50438695017059 0loss/FRt2/assignment_loss‚1.1780830698135571 0loss/FRt4/assignment_loss‚2.2661130568560433 .loss/FRt1/detection_loss‚1.748107379540615 .loss/FRt4/detection_loss‚3.435303014867446 _step‚966 1loss/SRqqt4/detection_loss‚2.8059141508170535 epoch‚9 % _timestamp‚1.7885241127013347e+09 2loss/SRqqt1/assignment_loss‚0.8817929527722299 2loss/SRqqt2/assignment_loss‚0.7935960717956748 2loss/SRqqt3/assignment_loss‚0.9676659725916882 /loss/FRt3/detection_loss‚2.2530659404159947 /loss/FBt2/detection_loss‚0.5100896450678339 1loss/FBt2/assignment_loss‚0.06896007431835673 1loss/FBt4/assignment_loss‚0.07228031336890126 /loss/FRt2/detection_loss‚2.1113207267167478 1loss/SRqqt1/detection_loss‚1.7648715823042123 _runtime‚13738.171455746 0loss/FRt1/assignment_loss‚2.4042266742749647 /loss/FRt3/assignment_loss‚2.035198963501237 2loss/FBt3/assignment_loss‚0.038439213199519776Æ[¢)”ð.‚(:‰ ÒìêÔªõèý gpu.0.gpu‚30.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚161.587' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚40.39675 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ž¯Gpñ.‚(:å ÒìêÔî¯Ù network.sent‚ 3.0907845e+07 disk.nvme1n1.in‚ 498.80078125' proc.memory.availableMB‚ 1293.828125 disk.rbd2.in‚0 disk.md0p1.in‚17.8125 disk.sda.in‚ 10.7578125 disk.rbd0.in‚0$ memory_percent‚92.10309982299805 network.recv‚ 4.392023e+06 disk.sda1.in‚0 disk.md0p1.out‚ 1514.16796875" disk.sda2.out‚1570.65673828125 cpu‚25.20555095406203% disk./.usageGB‚2713.8555221557617 disk.sda2.in‚ 10.7578125 disk.md0.in‚17.8125 disk.rbd2.out‚ 3.87109375) disk./.usagePercent‚75.88053623625188! disk.dm-0.out‚158128.98046875 disk.sdb.in‚ 7.0546875! disk.sda.out‚1570.65673828125 disk.rbd0.out‚ 3.87109375 disk.sdb1.out‚0) proc.memory.percent‚18.26450824737549 disk.sda1.out‚0 disk.dm-0.in‚ 1118.4375# disk.nvme0n1.out‚89387.66796875 disk.sdb1.in‚0 disk.md0.out‚ 1514.16796875" disk.sdb2.out‚1570.65673828125! disk.sdb.out‚1570.65673828125 disk.sdb2.in‚ 7.0546875 disk.nvme0n1.in‚ 619.63671875 disk.nvme1n1.out‚ 68741.3125# proc.memory.rssMB‚ 2992.45703125 proc.cpu.threads‚17·"lò.‚(:á áìêÔ¸ƒ€€" disk.nvme0n1.out‚ 89597.3828125 disk.sda2.in‚ 10.7578125 disk.nvme1n1.in‚ 498.80078125# disk.nvme1n1.out‚68784.96484375 disk.md0.out‚ 1515.640625 network.sent‚ 3.0973815e+07 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-0.in‚ 1118.4375 disk.md0p1.in‚17.8125 disk.sda.in‚ 10.7578125 disk.sda.out‚1572.1923828125! disk.sda2.out‚1572.1923828125 disk.rbd0.in‚0( proc.memory.availableMB‚ 1294.0859375 cpu‚25.20531564077228 disk.rbd0.out‚ 3.87109375 disk.rbd2.out‚ 3.87109375) disk./.usagePercent‚75.86578977772287! disk.dm-0.out‚158382.34765625 disk.sdb.in‚ 7.0546875 disk.sdb.out‚1572.1923828125# proc.memory.rssMB‚ 2992.45703125) proc.memory.percent‚18.26450824737549 disk.sdb1.out‚0 network.recv‚ 4.396393e+06% disk./.usageGB‚2713.3281173706055 disk.sdb1.in‚0 disk.md0p1.out‚ 1515.640625 disk.md0.in‚17.8125 disk.sdb2.in‚ 7.0546875 disk.nvme0n1.in‚ 619.63671875 disk.rbd2.in‚0! disk.sdb2.out‚1572.1923828125$ memory_percent‚92.10152626037598 proc.cpu.threads‚17Y¸¬’ó.‚(:‡ áìêÔÍîì€ gpu.0.gpu‚30.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚163.09' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚40.7725 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0+Y98“ô.‚(:ˆ ðìêÔÕÁÇÿ gpu.0.gpu‚46.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚37.0 gpu.0.powerWatts‚158.586' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚39.6465 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0s¨;[^õ.‚(:Ó ðìêÔ¢¶‹% disk./.usageGB‚2713.2832984924316 disk.sda2.out‚ 1573.7578125 disk.sda.out‚ 1573.7578125 network.recv‚ 4.40105e+06) disk./.usagePercent‚75.86453662313096 disk.rbd0.out‚ 3.87109375 disk.rbd2.out‚ 3.87109375 disk.dm-0.in‚ 1118.4375 disk.dm-0.out‚158540.6328125 disk.md0p1.out‚ 1517.15625 proc.cpu.threads‚17 network.sent‚ 3.0982079e+07# proc.memory.rssMB‚ 2992.45703125) proc.memory.percent‚18.26450824737549 disk.sda2.in‚ 10.7578125 disk.sdb1.out‚0 disk.sdb2.out‚ 1573.7578125 disk.sda1.out‚0 disk.sdb1.in‚0 disk.md0.out‚ 1517.15625 disk.sda1.in‚0 disk.md0p1.in‚17.8125) proc.memory.availableMB‚ 1294.11328125 disk.rbd0.in‚0 disk.nvme1n1.in‚ 498.80078125# disk.nvme1n1.out‚68785.86328125# disk.nvme0n1.out‚89754.76953125 disk.sdb.out‚ 1573.7578125# memory_percent‚92.1013593673706 disk.md0.in‚17.8125 disk.rbd2.in‚0 disk.sdb2.in‚ 7.0546875 disk.nvme0n1.in‚ 619.63671875 disk.sdb.in‚ 7.0546875 disk.sda.in‚ 10.7578125 cpu‚25.205106321394027È8c”ö.‚(:‰ ÿìêÔ¥žîÿ gpu.0.gpu‚26.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚36.0 gpu.0.powerWatts‚114.131' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.53275 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0©é^÷.‚(:Ó ÿìêÔ—ý³ disk.rbd2.out‚ 3.87109375! memory_percent‚92.10205078125) proc.memory.percent‚18.26450824737549 network.sent‚ 3.098568e+07% disk./.usageGB‚2713.3555068969727 disk.sdb.in‚ 7.0546875! disk.sdb.out‚1575.23876953125 disk.sda2.in‚ 10.7578125" disk.sda2.out‚1575.23876953125 disk.nvme1n1.in‚ 498.80078125# disk.nvme1n1.out‚68785.98828125 disk.sda1.out‚0 disk.md0p1.out‚ 1518.5703125 disk.dm-0.in‚ 1118.4375 disk.rbd0.in‚0 disk.rbd0.out‚ 3.87109375 disk.sdb2.in‚ 7.0546875 disk.sda1.in‚0# disk.nvme0n1.out‚89946.51171875" disk.sdb2.out‚1575.23876953125 disk.dm-0.out‚158732.5 cpu‚25.2049004276335( disk./.usagePercent‚75.8665556003435 disk.md0.in‚17.8125 disk.rbd2.in‚0 proc.memory.availableMB‚1294# proc.memory.rssMB‚ 2992.45703125 disk.md0p1.in‚17.8125! disk.sda.out‚1575.23876953125 disk.sdb1.out‚0 proc.cpu.threads‚17 network.recv‚ 4.405096e+06 disk.nvme0n1.in‚ 619.63671875 disk.sda.in‚ 10.7578125 disk.sdb1.in‚0 disk.md0.out‚ 1518.5703125t-c4ø./ _wandbruntime‚13795  _runtime‚13795§ø ±Ëù.Å ' lr-AdamW/pg1‚1.5127540832807524e-06 ' lr-AdamW/pg2‚1.5127540832807524e-06 trainer/global_step‚11799 % _timestamp‚1.7885241699812982e+09 _runtime‚13795.451618895 _step‚967%Ú`ÊÐú.Ê ' lr-AdamW/pg1‚1.5127540832807524e-06 ' lr-AdamW/pg2‚1.5127540832807524e-06 trainer/global_step‚11799 % _timestamp‚1.7885241699812982e+09 _runtime‚13795.451618895 _step‚967Ç€?Q–4û./ _wandbruntime‚13795  _runtime‚13795ò| ² (ü.…P M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955591168 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚620 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚618 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007241 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 % _timestamp‚1.7885241699823534e+09 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 trainer/global_step‚11799 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 295972944 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 151395333 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 401960960 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407872 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1212 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214665 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26090 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚983449095424512 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0úø¥Ä(ý.ŠP G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955591168 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚620 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚618 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007241 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 % _timestamp‚1.7885241699823534e+09 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 trainer/global_step‚11799 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 295972944 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 151395333 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 401960960 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407872 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1212 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214665 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26090 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚983449095424512 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090È„Ýa 4þ./ _wandbruntime‚13796  _runtime‚13796Šri5ÿ.N C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917424128 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 151407711 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955591168 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 222183648 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚961847609002270 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚970117435015168 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899421184 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚948484308859616 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚948484291820256 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899421184 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1085516273795584 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14007672268288 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 151407709 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14007896920576 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.pd&”eak‚607 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 224652288 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 243488446 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26114 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26114 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31686 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1099524194656256 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 243488446 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚70775939 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1099523946063872 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411628 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 539486007 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 539459891 trainer/global_step‚11799 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 295997561 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 401960960 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 _runtime‚13796.569938787 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚609 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 248592384 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚983529838523392 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 539459891 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177047040 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177047040 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1085516297735680 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚970117435015168 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13413302929408 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450988 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚983530755947520 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 539486007 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0U¦™÷'€/”N dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1085516297735680 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚970117435015168 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13413302929408 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450988 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚983530755947520 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 539486007 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917424128 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 151407711 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955591168 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 222183648 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚961847609002270 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚970117435015168 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899421184 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚948484308859616 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚948484291820256 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 899421184 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1085516273795584 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14007672268288 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 151407709 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14007896920576 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 224652288 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 243488446 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26114 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26114 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31686 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1099524194656256 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 243488446 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚70775939 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1099523946063872 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411628 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 539486007 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 539459891 trainer/global_step‚11799 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 295997561 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 401960960 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 _runtime‚13796.569938787 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚609 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 248592384 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚983529838523392 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 539459891 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177047040 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177047040És’õ14// _wandbruntime‚13796  _runtime‚13796\IÁà_‚/Ù 0loss/FRt3/assignment_loss‚2.0863912713770962 /loss/FRt4/assignment_loss‚2.605761828025182 1loss/FBt2/assignment_loss‚0.10237752610244291 /loss/FRt1/detection_loss‚2.0150839598512045 /loss/FRt4/detection_loss‚2.9214934607346854 .loss/FBt1/detection_loss‚2.543584108352661 .loss/FBt3/detection_loss‚1.855012704928716 .loss/FBt4/detection_loss‚2.909721723624638 1loss/FBt1/assignment_loss‚0.18909623291801836 &loss/total_loss‚36.514191981736374 trainer/global_step‚11799 .loss/FRt1/assignment_loss‚2.56219094881305 1loss/FBt3/assignment_loss‚0.19348050824575896 1loss/FBt4/assignment_loss‚0.10188130895500892 /loss/FRt2/detection_loss‚1.4148065923816628 1loss/SRqqt2/detection_loss‚1.2020284920930862 /loss/FBt2/detection_loss‚0.5651557141431619 /loss/FRt3/detection_loss‚1.9785265609317897 0loss/FRt2/assignment_loss‚1.4325196654708297 2loss/SRqqt1/assignment_loss‚1.1383520138365302 1loss/SRqqt3/detection_loss‚1.4832907669246196 1loss/SRqqt4/detection_loss‚2.1539805163355434 % _timestamp‚1.7885241711016853e+09 _step‚970 2loss/SRqqt2/assignment_loss‚0.9735352747565643 2loss/SRqqt3/assignment_loss‚1.2278678187931127 1loss/SRqqt4/assignment_loss‚1.434032801867408 epoch‚9 1loss/SRqqt1/detection_loss‚1.4240201822736047 _runtime‚13796.571816536d…Îdƒ/Þ /loss/FRt3/detection_loss‚1.9785265609317897 0loss/FRt2/assignment_loss‚1.4325196654708297 2loss/SRqqt1/assignment_loss‚1.1383520138365302 1loss/SRqqt3/detection_loss‚1.4832907669246196 1loss/SRqqt4/detection_loss‚2.1539805163355434 % _timestamp‚1.7885241711016853e+09 _step‚970 2loss/SRqqt2/assignment_loss‚0.9735352747565643 2loss/SRqqt3/assignment_loss‚1.2278678187931127 1loss/SRqqt4/assignment_loss‚1.434032801867408 epoch‚9 1loss/SRqqt1/detection_loss‚1.4240201822736047 _runtime‚13796.571816536 0loss/FRt3/assignment_loss‚2.0863912713770962 /loss/FRt4/assignment_loss‚2.605761828025182 1loss/FBt2/assignment_loss‚0.10237752610244291 /loss/FRt1/detection_loss‚2.0150839598512045 /loss/FRt4/detection_loss‚2.9214934607346854 .loss/FBt1/detection_loss‚2.543584108352661 .loss/FBt3/detection_loss‚1.855012704928716 .loss/FBt4/detection_loss‚2.909721723624638 1loss/FBt1/assignment_loss‚0.18909623291801836 &loss/total_loss‚36.514191981736374 trainer/global_step‚11799 .loss/FRt1/assignment_loss‚2.56219094881305 1loss/FBt3/assignment_loss‚0.19348050824575896 1loss/FBt4/assignment_loss‚0.10188130895500892 /loss/FRt2/detection_loss‚1.4148065923816628 1loss/SRqqt2/detection_loss‚1.2020284920930862 /loss/FBt2/detection_loss‚0.5651557141431619ÊÅ”7¿”„/‚(:‰ ŽíêÔð¾î€ gpu.0.gpu‚40.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚36.0 gpu.0.powerWatts‚117.193' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚29.29825 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Q•‹Ûl…/‚(:á ŽíêÔÛ¢Ÿ‚ disk.nvme1n1.in‚ 498.80078125 disk.sdb1.in‚0 disk.rbd2.out‚ 3.87109375 disk.dm-0.in‚ 1118.4375' proc.memory.availableMB‚ 1294.046875# proc.memory.rssMB‚ 2992.45703125 cpu‚25.20469300495941) disk./.usagePercent‚75.87004926607315# disk.nvme1n1.out‚68786.36328125 disk.sda1.in‚0 disk.md0p1.in‚17.8125# disk.nvme0n1.out‚90121.62890625 network.sent‚ 3.1004801e+07 disk.sdb1.out‚0 disk.md0.out‚ 1520.40625! disk.sdb.out‚1577.13427734375 disk.sda.in‚ 10.7578125 disk.md0.in‚17.8125 disk.rbd2.in‚0 disk.sda1.out‚0) proc.memory.percent‚18.26450824737549 disk.sdb2.in‚ 7.0546875 disk.dm-0.out‚158903.1171875 disk.sdb.in‚ 7.0546875$ disk./.usageGB‚2713.480457305908 disk.rbd0.out‚ 3.87109375 disk.nvme0n1.in‚ 619.63671875! disk.sda.out‚1577.13427734375" disk.sda2.out‚1577.13427734375 disk.rbd0.in‚0" disk.sdb2.out‚1577.13427734375 disk.md0p1.out‚ 1520.40625 proc.cpu.threads‚17 network.recv‚ 4.409587e+06 disk.sda2.in‚ 10.7578125$ memory_percent‚92.10176467895508d@Sd†/‚(:Ù íêÔ…—èÿ disk.sda1.out‚0 disk.dm-0.out‚ 159242.0625 disk.sda.out‚1578.669921875' proc.memory.availableMB‚ 1294.046875 disk.sdb2.in‚ 7.0546875 disk.sda1.in‚0! disk.nvme1n1.out‚ 68786.515625 network.sent‚ 3.1098993e+07 network.recv‚ 4.414184e+06) proc.memory.percent‚18.26450824737549 proc.cpu.threads‚17 disk.md0p1.in‚17.8125 disk.md0p1.out‚ 1521.88671875$ memory_percent‚92.10176467895508 disk.rbd2.in‚0 disk.nvme0n1.in‚ 619.63671875 disk.sdb.out‚1578.669921875 disk.nvme1n1.in‚ 498.80078125$ disk./.usageGB‚2713.605400085449 disk.sdb1.in‚0 disk.rbd2.out‚ 3.87109375 disk.sdb2.out‚1578.669921875 disk.dm-0.in‚ 1118.4375 disk.sda2.in‚ 10.7578125) disk./.usagePercent‚75.87354271848173 disk.md0.in‚17.8125! disk.nvme0n1.out‚ 90455.546875 disk.sdb.in‚ 7.0546875 disk.rbd0.in‚0# proc.memory.rssMB‚ 2992.45703125 cpu‚25.204480556043478 disk.sda.in‚ 10.7578125 disk.rbd0.out‚ 3.87109375 disk.sda2.out‚1578.669921875 disk.sdb1.out‚0 disk.md0.out‚ 1521.88671875®Ÿ³5”‡/‚(:‰ íêÔª¸Äÿ gpu.0.gpu‚47.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚37.0 gpu.0.powerWatts‚162.481' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚40.62025 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Þ‘cº”ˆ/‚(:‰ ¬íêÔåÿÑÿ gpu.0.gpu‚55.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚160.143' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚40.03575 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0'’aÌm‰/‚(:â ¬íêÔŠæèÿ network.sent‚ 3.1107405e+07! disk.sda.out‚1580.73291015625 disk.sda2.in‚ 10.7578125) proc.memory.availableMB‚ 1293.92578125! disk.nvme1n1.out‚ 68786.859375 disk.md0.out‚ 1523.87890625 network.recv‚ 4.418474e+06 disk.md0p1.in‚17.8125 disk.md0p1.out‚ 1523.87890625 disk.sdb.in‚ 7.0546875 disk.sda1.in‚0! disk.nvme0n1.out‚ 90649.203125 disk.sda1.out‚0 disk.rbd0.in‚0 disk.md0.in‚17.8125) proc.memory.percent‚18.26450824737549 proc.cpu.threads‚17 disk.rbd2.out‚ 3.87109375% disk./.usageGB‚2713.8545303344727 disk.dm-0.out‚ 159436.0625 disk.rbd0.out‚ 3.87109375 disk.nvme1n1.in‚ 498.80078125 disk.sdb1.out‚0 disk.sdb1.in‚0 disk.sdb2.in‚ 7.0546875 disk.sda.in‚ 10.7578125 disk.dm-0.in‚ 1118.4375 disk.nvme0n1.in‚ 619.63671875! disk.sdb.out‚1580.73291015625 cpu‚25.20426195576157 disk.rbd2.in‚0# proc.memory.rssMB‚ 2992.45703125) disk./.usagePercent‚75.88050850451353" disk.sda2.out‚1580.73291015625" disk.sdb2.out‚1580.73291015625# memory_percent‚92.1025037765503°78™pŠ/‚(:å »íêÔ—¯¨ÿ cpu‚25.204040013989854 disk.sdb2.in‚ 7.0546875 disk.dm-0.in‚ 1118.4375 disk.nvme0n1.in‚ 619.63671875 disk.sda.in‚ 10.7578125# proc.memory.rssMB‚ 2992.45703125) disk./.usagePercent‚75.88048493253594" disk.sdb2.out‚1582.43310546875! disk.sda.out‚1582.43310546875" disk.sda2.out‚1582.43310546875 disk.nvme1n1.in‚ 498.80078125' proc.memory.availableMB‚ 1293.921875$ disk./.usageGB‚2713.853687286377 disk.rbd2.in‚0 disk.dm-0.out‚159659.7578125 disk.sdb.in‚ 7.0546875! disk.sdb.out‚1582.43310546875 disk.md0.out‚ 1525.5234375 disk.rbd0.out‚ 3.87109375 disk.sdb1.out‚0# memory_percent‚92.1025276184082) proc.memory.percent‚18.26450824737549 network.recv‚ 4.422729e+06 disk.md0p1.out‚ 1525.5234375# disk.nvme0n1.out‚90872.74609375 disk.rbd0.in‚0 disk.md0.in‚17.8125 network.sent‚ 3.1113388e+07 disk.rbd2.out‚ 3.87109375 disk.sda1.out‚0 disk.md0p1.in‚17.8125# disk.nvme1n1.out‚68787.01171875 disk.sdb1.in‚0 disk.sda1.in‚0 disk.sda2.in‚ 10.7578125 proc.cpu.threads‚17ÏH×-”‹/‚(:‰ »íêÔè–ÿþ gpu.0.gpu‚14.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚178.629' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚44.65725 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0&Q¼4Œ// _wandbruntime‚13854  _runtime‚13854Y”+Ë/Å _runtime‚13854.233840824 _step‚971 ' lr-AdamW/pg1‚1.0377594560098268e-06 ' lr-AdamW/pg2‚1.0377594560098268e-06 trainer/global_step‚11849 % _timestamp‚1.7885242287632132e+09!ŒÄ¯ÐŽ/Ê ' lr-AdamW/pg2‚1.0377594560098268e-06 trainer/global_step‚11849 % _timestamp‚1.7885242287632132e+09 _runtime‚13854.233840824 _step‚971 ' lr-AdamW/pg1‚1.0377594560098268e-06Ëï4c94// _wandbruntime‚13854  _runtime‚13854j&Ý/…P TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13466945285120 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 223070848 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407872 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1103957578007552 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14064662395904 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411420 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 244463448 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚974019337500160 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407872 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 244463446 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1089892939551744 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26090 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 297179758 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚71056715 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚626 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1089892915611648 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 216276992 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 trainer/global_step‚11849 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚987487182193152 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚974019319497216 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1187 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 297179758 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917410816 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚974019337500160 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 297205846 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13467844692992 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 541643204 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚987486264782336 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26088 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955591168 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚987486264782336 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 244463446 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 152014133 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26088 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚987487182193152 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_traiWS1. n_batch_start/active_bytes.large_pool.peak‚ 12835443712 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13467844692992 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚952305333471888 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚952305316432528 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 244463448 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13417638138236 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚974019319497216 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 $ _timestamp‚1.788524228764504e+09 _runtime‚13854.235446142 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177047040 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214665 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 297205846 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚965722971610124 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0¼ÿ-/(‘/ŠP SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 401960960 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14064878672896 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚626 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1089892915611648 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 216276992 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 trainer/global_step‚11849 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚987487182193152 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚974019319497216 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1187 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 297179758 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917410816 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚974019337500160 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 297205846 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13467844692992 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 541643204 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚987486264782336 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26088 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955591168 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚987486264782336 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 244463446 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 152014133 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26088 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚987487182193152 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13467844692992 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚952305333471888 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚952305316432528 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 244463448 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13417638138236 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚974019319497216 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 $ _timestamp‚1.788524228764504e+09 _runtime‚13854.235446142 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177047040 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214665 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 297205846 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚965722971610124 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13466945285120 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 223070848 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407872 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1103957578007552 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14064662395904 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411420 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 244463448 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚974019337500160 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407872 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 244463446 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1089892939551744 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26090 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 297179758 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚71056715 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 223088947 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450988 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 541713809 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14066058867712 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411628 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26114 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177047040 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1104048317640192 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 231815168 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13417849388134 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚71062438 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1089982258772480 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 401960960 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚987567925305344 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1187 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26114 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1104048085825024 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 541687693 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _runtime‚13855.209923338 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177047040 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 541713809 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 152026509 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚614 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 207875072 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 _step‚973 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚965802060365778 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚952383313566016 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚952383296526656 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 $ _timestamp‚1.788524229739164e+09 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚987567007881216 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚974098949792768 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 297204349 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚616 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31686 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13418746799762]™ÕV$”/”N `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1089982234832384 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 trainer/global_step‚11849 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13468957509632 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13468957509632 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 541687693 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13418746799762 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 244483346 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚974098967795712 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 223088947 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450988 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 541713809 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14066058867712 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411628 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26114 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177047040 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1104048317640192 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 231815168 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13417849388134 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚71062438 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1089982258772480 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 401960960 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚987567925305344 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1187 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26114 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1104048085825024 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 541687693 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _runtime‚13855.209923338 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177047040 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 541713809 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 152026509 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚614 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 207875072 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 _step‚973 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚965802060365778 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚952383313566016 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚952383296526656 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 $ _timestamp‚1.788524229739164e+09 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚987567007881216 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚974098949792768 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 297204349 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚616 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PDeviceStatsMonitor.on_train3WaÑÄ_batch_end/active_bytes.all.current‚ 917424128 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31686 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 F0S\–/Ö 1loss/FBt3/assignment_loss‚0.06953719809109771 0loss/FBt4/assignment_loss‚0.6209685787725397 -loss/FBt4/detection_loss‚2.94768934779697 _runtime‚13855.211544275 .loss/FRt2/detection_loss‚1.463754663401741 .loss/FRt3/detection_loss‚2.559581214036697 1loss/SRqqt2/detection_loss‚0.9166767026504709 /loss/FBt1/detection_loss‚1.7593066096305847 2loss/SRqqt2/assignment_loss‚0.7575213624594552 0loss/SRqqt3/detection_loss‚1.664856170272005 1loss/SRqqt4/detection_loss‚2.5895770167311034 .loss/FBt2/detection_loss‚0.564696519113322 trainer/global_step‚11849 % _timestamp‚1.7885242297414153e+09 1loss/SRqqt1/assignment_loss‚1.286117218600379 2loss/SRqqt3/assignment_loss‚1.1797079070117966 1loss/FBt1/assignment_loss‚0.23256882385780955 &loss/total_loss‚38.320232019953394 _step‚974 /loss/FRt4/assignment_loss‚2.788056030869484 /loss/FRt4/detection_loss‚3.0154681596904993 /loss/FBt3/detection_loss‚2.0688123454650245 0loss/FRt2/assignment_loss‚1.0014682664709575 /loss/FRt3/assignment_loss‚2.540231856016012 1loss/SRqqt1/detection_loss‚1.9361169992221727 1loss/FBt2/assignment_loss‚0.12265221850725577 /loss/FRt1/detection_loss‚1.8910593577620947 epoch‚9 0loss/FRt1/assignment_loss‚3.0014438010611624 2loss/SRqqt4/assignment_loss‚1.34236365246276072„P*a—/Û /loss/FRt4/assignment_loss‚2.788056030869484 /loss/FRt4/detection_loss‚3.0154681596904993 /loss/FBt3/detection_loss‚2.0688123454650245 0loss/FRt2/assignment_loss‚1.0014682664709575 /loss/FRt3/assignment_loss‚2.540231856016012 1loss/SRqqt1/detection_loss‚1.9361169992221727 1loss/FBt2/assignment_loss‚0.12265221850725577 /loss/FRt1/detection_loss‚1.8910593577620947 epoch‚9 0loss/FRt1/assignment_loss‚3.0014438010611624 2loss/SRqqt4/assignment_loss‚1.3423636524627607 1loss/FBt3/assignment_loss‚0.06953719809109771 0loss/FBt4/assignment_loss‚0.6209685787725397 -loss/FBt4/detection_loss‚2.94768934779697 _runtime‚13855.211544275 .loss/FRt2/detection_loss‚1.463754663401741 .loss/FRt3/detection_loss‚2.559581214036697 1loss/SRqqt2/detection_loss‚0.9166767026504709 /loss/FBt1/detection_loss‚1.7593066096305847 2loss/SRqqt2/assignment_loss‚0.7575213624594552 0loss/SRqqt3/detection_loss‚1.664856170272005 1loss/SRqqt4/detection_loss‚2.5895770167311034 .loss/FBt2/detection_loss‚0.564696519113322 trainer/global_step‚11849 % _timestamp‚1.7885242297414153e+09 1loss/SRqqt1/assignment_loss‚1.286117218600379 2loss/SRqqt3/assignment_loss‚1.1797079070117966 1loss/FBt1/assignment_loss‚0.23256882385780955 &loss/total_loss‚38.320232019953394 _step‚974Ϋ[z’˜/‚(:‡ ÊíêԱ鑀 gpu.0.gpu‚53.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚37.0 gpu.0.powerWatts‚114.408' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚28.602 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‚2ªp™/‚(:å ÊíêÔƒ‡Î€ network.recv‚ 4.427531e+06" disk.sdb2.out‚1584.09716796875 disk.sda1.out‚0 disk.nvme0n1.out‚ 91212.46875 disk.sdb1.in‚0! disk.dm-0.out‚159999.70703125 disk.nvme0n1.in‚ 619.63671875" disk.sda2.out‚1584.09716796875 disk.nvme1n1.in‚ 498.80078125 disk.sdb1.out‚0 disk.md0.in‚17.8125) proc.memory.percent‚18.26450824737549% disk./.usageGB‚2713.8556365966797 disk.sdb2.in‚ 7.0546875 disk.sda1.in‚0& proc.memory.availableMB‚ 1293.46875 disk.sda2.in‚ 10.7578125 disk.rbd0.out‚ 3.87109375 disk.rbd2.in‚0) disk./.usagePercent‚75.88053943606785 disk.dm-0.in‚ 1118.4375# disk.nvme1n1.out‚68787.23828125$ memory_percent‚92.10529327392578 disk.md0p1.in‚17.8125! disk.sdb.out‚1584.09716796875 disk.rbd2.out‚ 3.87109375# proc.memory.rssMB‚ 2992.45703125 cpu‚25.20383266692244 disk.sda.in‚ 10.7578125! disk.sda.out‚1584.09716796875 disk.rbd0.in‚0 proc.cpu.threads‚17 network.sent‚ 3.1117463e+07 disk.md0p1.out‚ 1527.12109375 disk.sdb.in‚ 7.0546875 disk.md0.out‚ 1527.12109375º|•¨žš/‚(:“ ÙíêÔá–”€ gpu.0.gpu‚36.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚36.0 gpu.0.powerWatts‚126.497' gpu.0.enforcedPowerLimitWatts‚400.0) gpu.0.powerPercent‚31.624249999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÒÖl›/‚(:á ÙíêÔÂØõ disk.md0.in‚17.8125 disk.rbd2.out‚ 3.87109375) disk./.usagePercent‚75.86460403258727 disk.sdb1.in‚0! disk.sdb.out‚1585.48876953125 disk.sda.in‚ 10.7578125" disk.sda2.out‚1585.48876953125 disk.rbd0.in‚0 disk.nvme1n1.in‚ 498.80078125 disk.nvme1n1.out‚ 68787.40625 disk.md0p1.out‚ 1528.453125 disk.nvme0n1.in‚ 619.63671875! disk.sda.out‚1585.48876953125$ memory_percent‚92.10224151611328# proc.memory.rssMB‚ 2992.45703125 cpu‚25.20360772652391 network.recv‚ 4.432084e+06 disk.sdb1.out‚0" disk.sdb2.out‚1585.48876953125 disk.md0p1.in‚17.8125& proc.memory.availableMB‚ 1293.96875 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.in‚ 10.7578125 disk.md0.out‚ 1528.453125# disk.nvme0n1.out‚91381.94140625 network.sent‚ 3.1226722e+07 disk.rbd2.in‚0 disk.sdb2.in‚ 7.0546875 disk.sdb.in‚ 7.0546875) proc.memory.percent‚18.26450824737549 disk.dm-0.in‚ 1118.4375! disk.dm-0.out‚160169.34765625 disk.rbd0.out‚ 3.87109375 proc.cpu.threads‚17% disk./.usageGB‚2713.2857093811035½«”œ/‚(:‰ èíêÔœ¥Øÿ gpu.0.gpu‚48.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚124.321' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚31.08025 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‰†9 r/‚(:ç èíêÔæµöÿ disk.sdb1.out‚0 disk.md0p1.in‚17.8125) proc.memory.percent‚18.26450824737549 network.sent‚ 3.1232775e+07 disk.nvme0n1.in‚ 619.63671875 disk.sda.in‚ 10.7578125 disk.dm-0.out‚160331.2265625' proc.memory.availableMB‚ 1293.765625 disk.sdb2.in‚ 7.0546875 disk.sda1.in‚0 network.recv‚ 4.436739e+06 disk.md0.in‚17.8125 disk.md0.out‚ 1530.54296875 cpu‚25.20339585280998! disk.sda.out‚1587.66064453125 disk.nvme1n1.in‚ 498.80078125 disk.rbd2.out‚ 3.87109375 disk.sda1.out‚0$ memory_percent‚92.10348129272461 disk.rbd0.out‚ 3.87109375# disk.nvme1n1.out‚68787.76171875# proc.memory.rssMB‚ 2992.45703125$ disk./.usageGB‚2713.328197479248 disk.sda2.in‚ 10.7578125 disk.rbd0.in‚0 disk.sdb1.in‚0 disk.rbd2.in‚0) disk./.usagePercent‚75.86579201759403# disk.nvme0n1.out‚91543.46484375 disk.sdb.in‚ 7.0546875" disk.sda2.out‚1587.66064453125" disk.sdb2.out‚1587.66064453125 disk.dm-0.in‚ 1118.4375 disk.md0p1.out‚ 1530.54296875 proc.cpu.threads‚17! disk.sdb.out‚1587.66064453125 €L1“ž/‚(:ˆ ÷íêÔòÆÿ gpu.0.gpu‚47.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚165.266' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚41.3165 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0hðã`Ÿ/‚(:Õ ÷íêÔÿ²Ž disk.sda.in‚15.9375 proc.cpu.threads‚17 disk.md0p1.out‚ 1532.5703125 disk.sda2.in‚15.9375 disk.rbd2.in‚0" disk.sdb2.out‚1589.74951171875 disk.dm-0.in‚ 1118.4375" disk.sda2.out‚1589.74951171875 disk.rbd0.out‚ 3.87109375 network.sent‚ 3.1238695e+07 disk.sda1.in‚0 network.recv‚ 4.44074e+06 disk.rbd2.out‚ 3.87109375 disk.rbd0.in‚0# disk.nvme0n1.out‚91753.61328125) disk./.usagePercent‚75.86650909635149$ disk./.usageGB‚2713.353843688965# disk.nvme1n1.out‚68787.94921875 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.sda1.out‚0 cpu‚25.20319031371908 disk.nvme1n1.in‚ 498.80078125 disk.dm-0.out‚ 160541.5625 disk.md0p1.in‚27.625 disk.nvme0n1.in‚ 619.63671875 disk.sdb.in‚11.6875! disk.sdb.out‚1589.74951171875! disk.sda.out‚1589.74951171875$ memory_percent‚92.10348129272461' proc.memory.availableMB‚ 1293.765625# proc.memory.rssMB‚ 2992.45703125) proc.memory.percent‚18.26450824737549 disk.md0.in‚27.625 disk.md0.out‚ 1532.5703125 disk.sdb2.in‚11.6875³Á×±4 // _wandbruntime‚13911  _runtime‚13911Ü÷Ë É¡/à trainer/global_step‚11899 % _timestamp‚1.7885242864896686e+09 _runtime‚13911.960052677 _step‚975 & lr-AdamW/pg1‚6.519203210097917e-07 & lr-AdamW/pg2‚6.519203210097917e-07ŸÎ¢/È & lr-AdamW/pg2‚6.519203210097917e-07 trainer/global_step‚11899 % _timestamp‚1.7885242864896686e+09 _runtime‚13911.960052677 _step‚975 & lr-AdamW/pg1‚6.519203210097917e-07ÏGÓå44£// _wandbruntime‚13911  _runtime‚13911n½€û (¤/†P _runtime‚13911.961227672 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917410816 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚71344922 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26088 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955591168 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13522599865344 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13473084795344 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚71345534 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007241 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1212 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13523499273216 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13472187383924 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 245458348 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26088 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14123188589056 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 401960960 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚969676508522852 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚612 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚978000834274816 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 152632933 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 223977855 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚991523434140160 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407872 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 245458346 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 298438748 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚956204321138928 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1094358900588544 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214665 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26090 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1108482089177600 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14122959729152 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 _step‚976 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 298412660 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955591168 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13523499273216 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411420 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 228859904 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚991524351550976 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 223978469 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 trainer/global_step‚11899 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13522599865344 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 543897096 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚956204338178288 % _timestamp‚1.7885242864907982e+09 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450780 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 298412660 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚614 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚991524351550976 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚978000852277760 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 245458346 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177047040 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1108481836377600 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚969677422973632 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450780 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 298412660 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚614 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚991524351550976 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚978000852277760 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 245458346 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177047040 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚31686 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1108481836377600 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚969677422973632 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917410816 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚71344922 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26088 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955591168 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13522599865344 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13473084795344 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚71345534 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1175007241 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1212 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13523499273216 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13472187383924 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 245458348 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26088 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14123188589056 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 401960960 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚969676508522852 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚612 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚978000834274816 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 152632933 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 223977855 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚991523434140160 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 899407872 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 245458346 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 298438748 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚956204321138928 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1094358900588544 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214665 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26090 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1108482089177600 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14122959729152 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 _step‚976 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 298412660 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955591168 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13523499273216 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚35 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411420 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 228859904 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚991524351550976 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 223978469 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 trainer/global_step‚11899 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13522599865344 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 543897096 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚956204338178288 % _timestamp‚1.7885242864907982e+09 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 543941611 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13524612089856 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 543941611 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚969755597278298 P@DeviceStatsMÖJ|­«onitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1108572573344768 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917424128 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13474193456870 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚978080464570368 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 298463365 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚991604177239040 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚969756511729286 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007241 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 543915495 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 218360832 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚956282301233056 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177047040 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13523712668672 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 trainer/global_step‚11899 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚978080482573312 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚71351231 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450988 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 223995935 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚605 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955591168 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1108572331043840 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 298463365 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 152645309 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13523712668672 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177047040 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 242300928 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 % _timestamp‚1.7885242877922904e+09 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 543915495 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚991605094663168 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899421184 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 245478246 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1187 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1094448195869184 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14124353535488 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26114 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚978080482573312 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1094448219809280(Ö`'¨/”N ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13524612089856 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 543941611 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚969755597278298 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1108572573344768 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917424128 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13474193456870 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚978080464570368 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 298463365 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚991604177239040 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚969756511729286 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1175007241 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 543915495 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 218360832 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚956282301233056 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177047040 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13523712668672 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 trainer/global_step‚11899 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚978080482573312 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚71351231 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450988 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 223995935 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚605 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955591168 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1108572331043840 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 298463365 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 152645309 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13523712668672 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177047040 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚31931 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 242300928 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 % _timestamp‚1.7885242877922904e+09 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 543915495 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚991605094663168 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899421184 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 245478246 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1187 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1094448195869184 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14124353535488 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚26114 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚978080482573312 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1094448219809280 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚991605094663168 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 543941611 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564ѬÑa–4©// _wandbruntime‚13913  _runtime‚13913Å ÷\ª/Ö 0loss/FRt4/assignment_loss‚2.5950011881915005 0loss/FBt4/assignment_loss‚0.1453742738228243 /loss/FRt1/detection_loss‚2.0545062632160693 .loss/FBt1/detection_loss‚1.846161554257075 /loss/FBt4/detection_loss‚3.1908587962388992 _runtime‚13913.264982827 /loss/FRt2/assignment_loss‚1.135935554901759 1loss/SRqqt4/assignment_loss‚1.452721768631958 .loss/FRt2/detection_loss‚1.514343142664681 1loss/SRqqt2/assignment_loss‚0.837172584799451 1loss/SRqqt2/detection_loss‚1.1574668284239513 %loss/total_loss‚37.00857747960158 0loss/FRt3/assignment_loss‚2.4839762414202973 1loss/SRqqt3/assignment_loss‚1.119540688205968 0loss/FBt2/detection_loss‚0.44587742543538633 /loss/FRt1/assignment_loss‚2.390260312868201 1loss/FBt1/assignment_loss‚0.21845610450933842 0loss/FBt3/assignment_loss‚0.4563333114409228 /loss/FBt3/detection_loss‚2.0934006571769714 trainer/global_step‚11899 _step‚978 % _timestamp‚1.7885242877946725e+09 2loss/SRqqt1/assignment_loss‚1.0062512684613467 /loss/FRt4/detection_loss‚2.7106161496855994 0loss/SRqqt1/detection_loss‚1.624465530179441 1loss/SRqqt4/detection_loss‚2.5784997940063477 epoch‚9 1loss/FBt2/assignment_loss‚0.16254108996365507 /loss/FRt3/detection_loss‚2.2834232252310303 1loss/SRqqt3/detection_loss‚1.5053937258689027ªŒèa«/Û 1loss/SRqqt2/assignment_loss‚0.837172584799451 1loss/SRqqt2/detection_loss‚1.1574668284239513 %loss/total_loss‚37.00857747960158 0loss/FRt3/assignment_loss‚2.4839762414202973 1loss/SRqqt3/assignment_loss‚1.119540688205968 0loss/FBt2/detection_loss‚0.44587742543538633 /loss/FRt1/assignment_loss‚2.390260312868201 1loss/FBt1/assignment_loss‚0.21845610450933842 0loss/FBt3/assignment_loss‚0.4563333114409228 /loss/FBt3/detection_loss‚2.0934006571769714 trainer/global_step‚11899 _step‚978 % _timestamp‚1.7885242877946725e+09 2loss/SRqqt1/assignment_loss‚1.0062512684613467 /loss/FRt4/detection_loss‚2.7106161496855994 0loss/SRqqt1/detection_loss‚1.624465530179441 1loss/SRqqt4/detection_loss‚2.5784997940063477 epoch‚9 1loss/FBt2/assignment_loss‚0.16254108996365507 /loss/FRt3/detection_loss‚2.2834232252310303 1loss/SRqqt3/detection_loss‚1.5053937258689027 0loss/FRt4/assignment_loss‚2.5950011881915005 0loss/FBt4/assignment_loss‚0.1453742738228243 /loss/FRt1/detection_loss‚2.0545062632160693 .loss/FBt1/detection_loss‚1.846161554257075 /loss/FBt4/detection_loss‚3.1908587962388992 _runtime‚13913.264982827 /loss/FRt2/assignment_loss‚1.135935554901759 1loss/SRqqt4/assignment_loss‚1.452721768631958 .loss/FRt2/detection_loss‚1.514343142664681Ò¦{ñy’¬/‚(:‡ †îêÔþ™š gpu.0.gpu‚48.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚164.372' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚41.093 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÍÉwA_­/‚(:Ô †îêÔ­‹Ôƒ$ disk./.usageGB‚2713.355701446533 disk.sda.out‚1591.5205078125 disk.rbd2.out‚ 3.87109375( proc.memory.availableMB‚ 1293.7265625 disk.nvme0n1.in‚ 619.63671875 disk.md0.out‚ 1534.26953125$ memory_percent‚92.10371971130371) proc.memory.percent‚18.26450824737549 network.sent‚ 3.1255757e+07 disk.md0p1.out‚ 1534.26953125 disk.sdb1.in‚0 disk.rbd2.in‚0 disk.md0.in‚27.625) disk./.usagePercent‚75.86656104003063 disk.sda1.out‚0 disk.dm-0.in‚ 1118.4375 disk.sda2.in‚15.9375 disk.nvme1n1.in‚ 498.80078125# disk.nvme1n1.out‚68862.06640625 disk.sdb2.in‚11.6875 disk.md0p1.in‚27.625 disk.nvme0n1.out‚ 91858.625! disk.sda2.out‚1591.5205078125# proc.memory.rssMB‚ 2992.45703125 cpu‚25.20294431775385 disk.sda1.in‚0 disk.sdb.in‚11.6875 disk.sdb.out‚1591.5205078125 network.recv‚ 4.445748e+06! disk.dm-0.out‚160720.69140625 disk.rbd0.out‚ 3.87109375 proc.cpu.threads‚17! disk.sdb2.out‚1591.5205078125 disk.sda.in‚15.9375 disk.rbd0.in‚0 disk.sdb1.out‚0˹bC”®/‚(:‰ •îêÔð¡Úÿ gpu.0.gpu‚56.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚37.0 gpu.0.powerWatts‚116.583' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚29.14575 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0b¦ø÷`¯/‚(:Õ •îêÔÿøÙ€( disk./.usagePercent‚75.8700527858707 disk.dm-0.out‚161007.9609375! disk.sdb.out‚1593.13623046875$ memory_percent‚92.10517406463623# proc.memory.rssMB‚ 2992.45703125 network.recv‚ 4.450573e+06 disk.dm-0.in‚ 1118.4375# disk.nvme0n1.out‚92068.27734375 disk.rbd2.out‚ 3.87109375$ disk./.usageGB‚2713.480583190918 disk.md0p1.out‚ 1535.8125 disk.rbd0.out‚ 3.87109375 disk.md0.in‚27.625" disk.sdb2.out‚1593.13623046875) proc.memory.percent‚18.26450824737549 disk.sda.in‚15.9375 disk.sda2.in‚15.9375 disk.sdb1.out‚0 disk.sdb2.in‚11.6875 disk.sda1.out‚0 cpu‚25.202737293884855 network.sent‚ 3.1349964e+07 disk.rbd0.in‚0 disk.sdb1.in‚0 proc.cpu.threads‚17 disk.md0p1.in‚27.625 disk.sdb.in‚11.6875 disk.nvme1n1.in‚ 498.80078125) proc.memory.availableMB‚ 1293.48828125 disk.nvme0n1.in‚ 619.63671875! disk.sda.out‚1593.13623046875# disk.nvme1n1.out‚68939.68359375 disk.md0.out‚ 1535.8125 disk.sda1.in‚0" disk.sda2.out‚1593.13623046875 disk.rbd2.in‚06}M¯”°/‚(:‰ ¤îêÔÜ‚ˆ€ gpu.0.gpu‚55.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚170.047' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.51175 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÙÅ e±/‚(:Ú ¤îêÔÍÿ€ disk.md0.out‚ 1537.796875 disk.rbd2.in‚0 disk.sdb.in‚11.6875 proc.cpu.threads‚17# disk.nvme0n1.out‚92208.86328125) proc.memory.percent‚18.26450824737549 disk.rbd0.in‚0 disk.rbd0.out‚ 3.87109375! disk.sda.out‚1595.17626953125# disk.nvme1n1.out‚69012.64453125 disk.sdb2.in‚11.6875 disk.nvme0n1.in‚ 619.63671875 disk.dm-0.in‚ 1118.4375 disk.md0p1.out‚ 1537.796875 cpu‚25.202518588843652 disk.sda.in‚15.9375 disk.sdb1.in‚0 disk.sdb1.out‚0" disk.sdb2.out‚1595.17626953125# proc.memory.rssMB‚ 2992.45703125 disk.sda1.in‚0) proc.memory.availableMB‚ 1293.73828125 disk.md0.in‚27.625 disk.rbd2.out‚ 3.87109375 disk.md0p1.in‚27.625$ memory_percent‚92.10364818572998 network.sent‚ 3.1355078e+07$ disk./.usageGB‚2713.605815887451 disk.sda2.in‚15.9375 disk.nvme1n1.in‚ 498.80078125 disk.sda1.out‚0 disk.dm-0.out‚161221.5078125! disk.sdb.out‚1595.17626953125 network.recv‚ 4.454269e+06) disk./.usagePercent‚75.87355434447973" disk.sda2.out‚1595.176269531259Ÿ”²/‚(:‰ ³îêÔ¬€´ÿ gpu.0.gpu‚39.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚170.891' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚42.72275 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Y¾xZ³/‚(:Ï ³îêÔ™úÁ€ disk.md0.out‚ 1539.1640625 disk.sda1.in‚0 network.recv‚ 4.458455e+06 disk.rbd2.in‚0! disk.sdb2.out‚1596.5927734375 disk.md0p1.in‚27.625# proc.memory.rssMB‚ 2992.45703125) proc.memory.percent‚18.26450824737549) disk./.usagePercent‚75.88057292747493 disk.md0.in‚27.625 disk.rbd2.out‚ 3.87109375 disk.sdb2.in‚11.6875 disk.dm-0.in‚ 1118.4375 disk.sda.out‚1596.5927734375 disk.sda1.out‚0 disk.nvme0n1.in‚ 619.63671875# disk.nvme0n1.out‚92383.31640625 disk.sdb.in‚11.6875 disk.nvme1n1.in‚ 498.80078125 disk.nvme1n1.out‚69044 disk.sdb1.in‚0 disk.rbd0.out‚ 3.87109375 disk.rbd0.in‚0$ disk./.usageGB‚2713.856834411621 disk.md0p1.out‚ 1539.1640625 disk.sdb.out‚1596.5927734375 disk.sda.in‚15.9375! disk.sda2.out‚1596.5927734375 disk.sdb1.out‚0 disk.sda2.in‚15.9375) proc.memory.availableMB‚ 1293.73828125 network.sent‚ 3.1363074e+07! disk.dm-0.out‚161427.31640625$ memory_percent‚92.10364818572998 cpu‚25.20230324017957 proc.cpu.threads‚17E+‹;4´// _wandbruntime‚13969  _runtime‚13969ütð4ǵ/Á % _timestamp‚1.7885243443954036e+09 _runtime‚13969.865850376 _step‚979 % lr-AdamW/pg1‚3.55317086578187e-07 % lr-AdamW/pg2‚3.55317086578187e-07 trainer/global_step‚11949¢¥“ñ̶/Æ % lr-AdamW/pg1‚3.55317086578187e-07 % lr-AdamW/pg2‚3.55317086578187e-07 trainer/global_step‚11949 % _timestamp‚1.7885243443954036e+09 _runtime‚13969.865850376 _step‚979Ó±9¹¾4·// _wandbruntime‚13969  _runtime‚13969 ¦F¸/…P UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407872 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1212 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 153251735 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411420 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1187 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177047040 trainer/global_step‚11949 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13528531452452 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 _runtime‚13969.86676115 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 299671650 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚960103325845328 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚995560603497984 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 224883631 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1113006263687680 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚625 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13579153853440 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 299671650 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚973630959886360 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13527634041032 M=DeviceStatsMonitor.on_train_batch_star;ðWt/active_bytes.all.peak‚ 13955591168 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚981982349052416 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955591168 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214665 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 299645562 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1098824837685248 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 % _timestamp‚1.7885243443962352e+09 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 299645562 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13579153853440 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410816 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚981982349052416 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450780 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26088 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14181402062336 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917410816 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚995561520908800 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1113006017179136 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 222568448 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 401960960 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚981982367055360 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚71631896 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚995561520908800 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26090 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13578254445568 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 246453246 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚995560603497984 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 546124898 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 246453246 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 153251733 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚960103342884688 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚981982367055360 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 246453248 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360®² ½(¹/ŠP GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 299671650 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚973630959886360 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13527634041032 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955591168 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚981982349052416 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955591168 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1171 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214665 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 299645562 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1098824837685248 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 12392107008 % _timestamp‚1.7885243443962352e+09 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 299645562 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13579153853440 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410816 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚981982349052416 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450780 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚26088 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14181402062336 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917410816 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚995561520908800 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1113006017179136 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 222568448 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 401960960 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1182793728 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚607 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚981982367055360 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚71631896 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚995561520908800 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚26090 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13578254445568 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 246453246 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚995560603497984 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 546124898 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 246453246 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 153251733 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚960103342884688 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚981982367055360 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 246453248 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚3677 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407872 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1212 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚564 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 153251735 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411420 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1187 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177047040 trainer/global_step‚11949 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13528531452452 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚607 _runtime‚13969.86676115 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 299671650 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚960103325845328 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚995560603497984 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 224883631 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1113006263687680 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚625 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13579153853440ÔüìÀ4º// _wandbruntime‚13970  _runtime‚13970iû„©'»/N SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 246473146 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚973710048641806 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚981 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899421184 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1098914156905984 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14182555176448 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚973710963092794 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚960181322978816 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13529640113978 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955591168 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚643 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 546143297 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 299696267 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31686 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚71637528 trainer/global_step‚11949 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13580266670080 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 224900996 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 246473146 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 263272448 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1098914180846080 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13579367248896 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 299670153 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚982061997350912 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚995642264020992 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚995641346596864 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 _runtime‚13970.902077077 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚995642264020992 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 299670153 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13579367248896 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚982061997350912 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚982061979347968 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 401960960 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 546169413 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14182315844096 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1187 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 246473144 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177047040 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚981 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899421184 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1098914156905984 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14182555176448 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚973710963092794 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚960181322978816 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13529640113978 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955591168 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚643 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 546143297 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 299696267 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚31686 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚71637528 trainer/global_step‚11949 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13580266670080 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 224900996 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 246473146 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 263272448 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1098914180846080 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1182793728 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13579367248896 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 299670153 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚982061997350912 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚995642264020992 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚995641346596864 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 _runtime‚13970.902077077 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 C=DeviceStatsMonito;òž r.on_train_batch_end/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚995642264020992 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 299670153 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13579367248896 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚982061997350912 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚982061979347968 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 401960960 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 546169413 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14182315844096 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1187 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 246473144 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177047040 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450780 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14239674089472 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 247448146 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13634808433664 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚985963863830016 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚999597772855808 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚964002330551728 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 _runtime‚14027.644911869 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 300878464 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚625 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13633909025792 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410816 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 548326610 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 247448148 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13634808433664 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 300878464 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 401960960 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13583080698140 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955591168 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 300904552 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26088 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 153870533 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407872 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚999597772855808 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 548352700 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚71917697 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1103290822662144 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411420 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13583978109560 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917410816 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚999597772855808 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 548352700 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚71917697 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1103290822662144 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 897411420 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13583978109560 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 917410816 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450780 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14239674089472 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 247448146 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13634808433664 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚31931 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1171 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚985963863830016 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚999597772855808 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚964002330551728 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 _runtime‚14027.644911869 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 300878464 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚625 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1171 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13633909025792 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 917410816 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 548326610 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 247448148 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13634808433664 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 300878464 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 401960960 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13583080698140 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 13955591168 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 300904552 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26088 UGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚ 153870533 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 899407872 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 13037993984 JDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411628 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 153882909 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1187 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1103380117942784 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450988 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 247468046 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955591168 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26114 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1117620970198528 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 300929169 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚986043512128512 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899421184 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14240828315648 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚986043494125568 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 225805706 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177047040 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 247468044 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚624 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚977664500005314 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 548397215 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1117620715314688 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14240597371904 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 153882911 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1103380141882880 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 trainer/global_step‚11999 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚999678515954688 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 230943744 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917424128 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚986043512128512 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13635021829120 _runtime‚14028.592058375 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚986043494125568 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 225806330 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 401960960 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13635021829120 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 548371099 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 300929169 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 548371099 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13635921250304 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13584189359458 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0—ú8'Ð/•N XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 175228416 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177047040 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 247468044 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚624 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚977664500005314 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 548397215 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚3677 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1117620715314688 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14240597371904 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚564 WIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚ 153882911 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1103380141882880 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚607 trainer/global_step‚11999 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚999678515954688 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 230943744 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917424128 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1171 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚986043512128512 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13635021829120 _runtime‚14028.592058375 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 14220787712 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 13037993984 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚986043494125568 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 225806330 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 401960960 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13635021829120 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 548371099 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 300929169 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1171 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 548371099 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13635921250304 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13584189359458 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411628 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1171 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚ 153882909 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1187 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1103380117942784 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450988 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 12392107008 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 247468046 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955591168 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚26114 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1117620970198528 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 300929169 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚986043512128512 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 899421184 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14240828315648 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚986043494125568 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 12835443712 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚26116 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 225805706Ù½fPû4Ñ// _wandbruntime‚14028  _runtime‚14028lò„cÒ/Ý 1loss/SRqqt1/detection_loss‚1.7481749908245625 1loss/SRqqt4/detection_loss‚2.6961193405664883 epoch‚9 % _timestamp‚1.7885244031234365e+09 1loss/SRqqt3/detection_loss‚1.4710669801834058 .loss/FBt4/detection_loss‚3.447851371765137 1loss/SRqqt3/assignment_loss‚1.069690956628688 .loss/FRt3/detection_loss‚1.971088439799272 /loss/FBt2/detection_loss‚0.6673912836567444 1loss/FBt3/assignment_loss‚0.16537579828873278 /loss/FRt1/detection_loss‚2.0382045463485094 /loss/FBt1/detection_loss‚2.1252361701594458 .loss/FBt3/detection_loss‚2.489723014831543 _runtime‚14028.593552582 %loss/total_loss‚38.40808125537893 2loss/SRqqt2/assignment_loss‚0.9981796153821051 3loss/FBt1/assignment_loss‚0.0021901160555065267 /loss/FRt4/detection_loss‚2.7005928198496503 1loss/SRqqt2/detection_loss‚0.9160290881991386 trainer/global_step‚11999 /loss/FRt1/assignment_loss‚2.477715334226919 0loss/FRt3/assignment_loss‚2.6959510216346154 2loss/SRqqt1/assignment_loss‚0.9592367383909035 2loss/SRqqt4/assignment_loss‚1.5603156614188964 1loss/FBt2/assignment_loss‚0.15715209201574515 _step‚986 0loss/FRt2/assignment_loss‚1.3443294965451764 0loss/FRt4/assignment_loss‚3.2319264729817707 2loss/FBt4/assignment_loss‚0.015805974922841415 /loss/FRt2/detection_loss‚1.4587339307031324ɾIèhÓ/â 1loss/SRqqt3/assignment_loss‚1.069690956628688 .loss/FRt3/detection_loss‚1.971088439799272 /loss/FBt2/detection_loss‚0.6673912836567444 1loss/FBt3/assignment_loss‚0.16537579828873278 /loss/FRt1/detection_loss‚2.0382045463485094 /loss/FBt1/detection_loss‚2.1252361701594458 .loss/FBt3/detection_loss‚2.489723014831543 _runtime‚14028.593552582 %loss/total_loss‚38.40808125537893 2loss/SRqqt2/assignment_loss‚0.9981796153821051 3loss/FBt1/assignment_loss‚0.0021901160555065267 /loss/FRt4/detection_loss‚2.7005928198496503 1loss/SRqqt2/detection_loss‚0.9160290881991386 trainer/global_step‚11999 /loss/FRt1/assignment_loss‚2.477715334226919 0loss/FRt3/assignment_loss‚2.6959510216346154 2loss/SRqqt1/assignment_loss‚0.9592367383909035 2loss/SRqqt4/assignment_loss‚1.5603156614188964 1loss/FBt2/assignment_loss‚0.15715209201574515 _step‚986 0loss/FRt2/assignment_loss‚1.3443294965451764 0loss/FRt4/assignment_loss‚3.2319264729817707 2loss/FBt4/assignment_loss‚0.015805974922841415 /loss/FRt2/detection_loss‚1.4587339307031324 1loss/SRqqt1/detection_loss‚1.7481749908245625 1loss/SRqqt4/detection_loss‚2.6961193405664883 epoch‚9 % _timestamp‚1.7885244031234365e+09 1loss/SRqqt3/detection_loss‚1.4710669801834058 .loss/FBt4/detection_loss‚3.447851371765137ÚÈKe”Ô/‚(:‰ þîêÔ½¨ÿ gpu.0.gpu‚53.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚37.0 gpu.0.powerWatts‚124.321' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚31.08025 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0âüôeÕ/‚(:Ú þîêÔÁŒÝ€ disk.rbd2.in‚0 disk.dm-0.out‚ 162589.546875 disk.nvme0n1.in‚ 619.63671875 disk.rbd0.in‚0# proc.memory.rssMB‚ 2992.45703125 proc.cpu.threads‚17% disk./.usageGB‚2713.3538360595703 disk.sda1.out‚0# disk.nvme0n1.out‚93170.23828125 disk.sdb.in‚11.6875" disk.sda2.out‚1604.60009765625 disk.nvme1n1.in‚ 498.80078125 disk.md0.out‚ 1546.8515625 disk.sda1.in‚0! disk.sda.out‚1604.60009765625) proc.memory.percent‚18.26450824737549! disk.sdb.out‚1604.60009765625 disk.sda2.in‚15.9375 network.recv‚ 4.481264e+06 disk.rbd2.out‚ 3.87109375" disk.sdb2.out‚1604.60009765625 network.sent‚ 3.1505505e+07) disk./.usagePercent‚75.86650888303042 disk.sdb1.out‚0 disk.md0.in‚27.625 disk.dm-0.in‚ 1118.4375 disk.md0p1.in‚27.625) proc.memory.availableMB‚ 1293.38671875 cpu‚25.20133036230973 disk.sdb2.in‚11.6875 disk.sda.in‚15.9375 disk.rbd0.out‚ 3.87109375# disk.nvme1n1.out‚69419.30859375 disk.sdb1.in‚0 disk.md0p1.out‚ 1546.8515625# memory_percent‚92.1057939529419;Òn›”Ö/‚(:‰ ïêÔŠ¤£ÿ gpu.0.gpu‚38.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚38.0 gpu.0.powerWatts‚160.475' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚40.11875 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ƒ€¶b×/‚(:× ïêÔÕ´Õÿ disk.dm-0.out‚162681.7421875 disk.md0p1.out‚ 1548.69921875 disk.nvme0n1.in‚ 619.63671875 disk.md0.out‚ 1548.69921875! disk.nvme0n1.out‚ 93234.296875 disk.nvme1n1.in‚ 498.80078125' proc.memory.availableMB‚ 1293.640625' disk./.usagePercent‚75.867432243257$ disk./.usageGB‚2713.386859893799! disk.sdb.out‚1606.48681640625# proc.memory.rssMB‚ 2992.45703125" disk.sda2.out‚1606.48681640625 disk.sdb1.out‚0) proc.memory.percent‚18.26450824737549 disk.rbd2.out‚ 3.87109375 disk.sdb2.in‚11.6875 disk.sda1.out‚0! disk.sda.out‚1606.48681640625 disk.rbd0.out‚ 3.87109375 proc.cpu.threads‚17 network.sent‚ 3.1601974e+07 disk.dm-0.in‚ 1118.4375 disk.sda.in‚15.9375 disk.sda2.in‚15.9375 disk.sdb1.in‚0 network.recv‚ 4.485797e+06 disk.md0.in‚27.625 disk.sda1.in‚0 disk.sdb.in‚11.6875 disk.rbd0.in‚0" disk.sdb2.out‚1606.48681640625 disk.md0p1.in‚27.625" disk.nvme1n1.out‚ 69447.4453125$ memory_percent‚92.10424423217773 cpu‚25.201155350225143 disk.rbd2.in‚0Š¯Zp”Ø/‚(:‰ œïêÔßÿ gpu.0.gpu‚52.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚17.198257446289062* gpu.0.memoryAllocatedBytes‚ 14773190656 gpu.0.temp‚37.0 gpu.0.powerWatts‚121.259' gpu.0.enforcedPowerLimitWatts‚400.0 gpu.0.powerPercent‚30.31475 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‚ý×gÙ/‚(:Ü œïêÔ­Ú disk.md0.out‚ 1550.06640625 disk.rbd2.out‚ 3.87109375 disk.sda1.in‚0 disk.rbd0.in‚0' proc.memory.availableMB‚ 1293.640625% disk./.usageGB‚2712.5524978637695 disk.md0.in‚27.625! disk.nvme0n1.out‚ 93380.109375! disk.sda.out‚1607.89697265625 disk.nvme1n1.in‚ 498.80078125 proc.cpu.threads‚17 disk.md0p1.out‚ 1550.06640625# proc.memory.rssMB‚ 2992.45703125) proc.memory.percent‚18.26450824737549 cpu‚25.200915950960358 disk.sdb2.in‚11.6875" disk.sda2.out‚1607.89697265625 disk.rbd0.out‚ 3.87109375# disk.nvme1n1.out‚69525.42578125 network.sent‚ 3.1605918e+07" disk.sdb2.out‚1607.89697265625 disk.nvme0n1.in‚ 619.63671875 disk.sdb.in‚11.6875! disk.sdb.out‚1607.89697265625 disk.sdb1.in‚0 disk.sdb1.out‚0 disk.rbd2.in‚0 disk.dm-0.in‚ 1118.4375 disk.md0p1.in‚27.625 disk.sda.in‚15.9375 network.recv‚ 4.490051e+06) disk./.usagePercent‚75.84410313168975$ memory_percent‚92.10424423217773 disk.sda1.out‚0! disk.dm-0.out‚162905.53515625 disk.sda2.in‚15.9375‹?Õ4Ú// _wandbruntime‚14082  _runtime‚14082ôUkgÊÛ/Ä ' lr-AdamW/pg1‚3.0046956588584484e-08 ' lr-AdamW/pg2‚3.0046956588584484e-08 trainer/global_step‚12049 $ _timestamp‚1.788524456754172e+09 _runtime‚14082.224501318 _step‚987õiÏÜ/É trainer/global_step‚12049 $ _timestamp‚1.788524456754172e+09 _runtime‚14082.224501318 _step‚987 ' lr-AdamW/pg1‚3.0046956588584484e-08 ' lr-AdamW/pg2‚3.0046956588584484e-08Û±ÁÎ4Ý// _wandbruntime‚14082  _runtime‚14082à:X‚;Þ/ŠP A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚31931 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1171 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 12835443712 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1003634942213632 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1122054362411520 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚981539862613376 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 550554412 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 226694493 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1107756759758848 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 154489335 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1212 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 226762752 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450780 _runtime‚14082.225397443 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1003635859624448 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1187 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 trainer/global_step‚12049 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 302137454 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955591168 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚72205786 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214665 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚989945396610560 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 401960960 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚967901352297488 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚967901335258128 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177047040 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 302137454 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26088 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 550580502 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7885244567549555e+09 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1003634942213632 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13690463013888 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13689563606016 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 248443046 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚26090 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 302137454 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 13955591168 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚72205786 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 13510214665 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚3677 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚989945396610560 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 12835443712 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 401960960 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚967901352297488 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚967901335258128 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1177047040 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 302137454 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 14220787712 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚607 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚26088 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 550580502 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚564 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 361446400 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 14220787712 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 14220787712 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7885244567549555e+09 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1003634942213632 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13690463013888 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13689563606016 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 248443046 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚31686 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 226694493 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1107756759758848 YKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚ 154489335 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚564 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1212 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 226762752 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 914450780 _runtime‚14082.225397443 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1003635859624448 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 OFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚1187 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 trainer/global_step‚12049 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 JDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 550598901 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚990025008903168 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1122144845250560 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚990025026906112 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 220457984 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1003716602736640 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177047040 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 226713219 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 401960960 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177047040 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13690676409344 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1003716602736640 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 226712591 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚72211508 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411628 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚628 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1187 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1122144600852480 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14298521872896 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13640533428194 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 550598901 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 % _timestamp‚1.7885244577132552e+09 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1003715685312512 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 248462946 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 550625017 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 302135957 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14298742330880 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚981619865819810 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917424128 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13691575830528 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚626 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450988 _runtime‚14083.184269121 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 302135957 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955591168™ï¾p¥â/™N L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 917424128 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 13955591168 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 550598901 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚990025008903168 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚31931 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1122144845250560 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚990025026906112 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 220457984 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1182793728 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1003716602736640 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1177047040 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 226713219 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 401960960 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1177047040 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1171 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13690676409344 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 361446400 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1003716602736640 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 226712591 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚607 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 13955591168 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚72211508 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 897411628 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚628 MDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚1187 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1122144600852480 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14298521872896 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13640533428194 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 550598901 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚35 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 14220787712 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 13037993984 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚564 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 13510214665 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚607 % _timestamp‚1.7885244577132552e+09 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1003715685312512 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 14220787712 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 248462946 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 550625017 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 302135957 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 12835443712 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14298742330880 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚981619865819810 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 917424128 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13691575830528 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚626 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚564 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1212 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 914450988 _runtime‚14083.184269121 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 302135957 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 BDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 OGDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.current‚564 NHDeviceStatsMonitor.on_validation_batch_start/oversize_segments.allocated‚0 `PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.freed‚0 QIDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.allocated‚564 [LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 TKDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.peak‚1187 OIDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.current‚0 \LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.peak‚ 12392107008 [LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.peak‚ 1175007241 F=DeviceStatsMonitor.on_validation_batch_start/segment.all.peak‚1171 PJDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_validation_batch_start/allocation.all.freed‚ 553563824 YIDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.allocated‚ 14220787712 I@DeviceStatsMonitor.on_validation_batch_start/segment.all.current‚1171 OGDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.current‚607 VHDeviceStatsMonitor.on_validation_batch_start/active.large_pool.allocated‚ 250007390 RDDeviceStatsMonitor.on_validation_batch_start/active.large_pool.freed‚ 250007388 ZLDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.allocated‚ 250007390 B;DeviceStatsMonitor.on_validation_batch_start/max_split_size‚-1 QIDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.allocated‚607 trainer/global_step‚12089 ZLDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.freed‚ 155459309 LFDeviceStatsMonitor.on_validation_batch_start/oversize_segments.current‚0 [KDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 PGDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.peak‚3677 kVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.allocated‚1114135895877120 dQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.allocated‚13760051065856 eQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.allocated‚973537270558112 KEDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.freed‚0 RBDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.peak‚ 13955591168 bNDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.allocated‚995835231241216 aNDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.allocated‚13760051065856 OADeviceStatsMonitor.on_validation_batch_start/active.all.allocated‚ 553584345 \ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.current‚18002944 `MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.freed‚13759375949312 SEDeviceStatsMonitor.on_validation_batch_start/allocation.all.allocated‚ 553584345 TNDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.current‚2 RKDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.peak‚35 `KDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.freed‚1128505269694464 aTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.current‚23940096 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.peak‚ 362095616 D>DeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 I?DeviceStatsMonitor.on_validation_batch_start/active.all.current‚20521 _JDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.allocated‚1009595282307072 ]ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.current‚ 673144760 _PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 dQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.allocated‚13708763727090 ZKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 J@DeviceStatsMonitor.on_validation_batch_start/allocation.all.peak‚31931 ^JDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.allocated‚987246034285202 `MDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.freed‚13708090582330 ^JDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.freed‚995835213238272 YIDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.peak‚ 12835443712 iVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.allocated‚14369540530176 F=DeviceStatsMonitor.on_validation_batch_start/num_device_alloc‚1171®Üí»ö/öT dQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.allocated‚13760051065856 eQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.allocated‚973537270558112 KEDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.freed‚0 RBDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.peak‚ 13955591168 bNDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.allocated‚995835231241216 aNDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.allocated‚13760051065856 OADeviceStatsMonitor.on_validation_batch_start/active.all.allocated‚ 553584345 \ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.current‚18002944 `MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.freed‚13759375949312 SEDeviceStatsMonitor.on_validation_batch_start/allocation.all.allocated‚ 553584345 TNDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.current‚2 RKDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.peak‚35 `KDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.freed‚1128505269694464 aTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.current‚23940096 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.peak‚ 362095616 D>DeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 I?DeviceStatsMonitor.on_validation_batch_start/active.all.current‚20521 _JDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.allocated‚1009595282307072 ]ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.current‚ 673144760 _PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.allocated‚ 1182793728 dQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.allocated‚13708763727090 ZKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.peak‚ 1182793728 J@DeviceStatsMonitor.on_validation_batch_start/allocation.all.peak‚31931 ^JDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.allocated‚987246034285202 `MDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.freed‚13708090582330 ^JDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.freed‚995835213238272 YIDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.peak‚ 12835443712 iVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.allocated‚14369540530176 F=DeviceStatsMonitor.on_validation_batch_start/num_device_alloc‚1171 E?DeviceStatsMonitor.on_validation_batch_start/num_oom_rejections‚0 TJDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.current‚20519 XJDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.peak‚ 401960960 ^NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.current‚ 13037993984 LDDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.peak‚607 % _timestamp‚1.7885245586886926e+09 K=DeviceStatsMonitor.on_validation_batch_start/active.all.freed‚ 553563824 aMDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.freed‚995835213238272 [MDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.current‚ 166712832 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.peak‚ 175228416 LCDeviceStatsMonitor.on_validation_batch_start/active.large_pool.peak‚3677 VHDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.current‚ 693119488 VHDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.freed‚ 250007388 FDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 OGDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.current‚564 NHDeviceStatsMonitor.on_validation_batch_start/oversize_segments.allocated‚0 `PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.allocated‚ 13037993984 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.freed‚0 QIDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.allocated‚564 [LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.peak‚ 1177047040 TKDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.peak‚1187 OIDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.current‚0 \LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.peak‚ 12392107008 [LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.peak‚ 1175007241 F=DeviceStatsMonitor.on_validation_batch_start/segment.all.peak‚1171 PJDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_validation_batch_start/allocation.all.freed‚ 553563824 YIDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.allocated‚ 14220787712 I@DeviceStatsMonitor.on_validation_batch_start/segment.all.current‚1171 OGDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.current‚607 VHDeviceStatsMonitor.on_validation_batch_start/active.large_pool.allocated‚ 250007390 RDDeviceStatsMonitor.on_validation_batch_start/active.large_pool.freed‚ 250007388 ZLDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.allocated‚ 250007390 B;DeviceStatsMonitor.on_validation_batch_start/max_split_size‚-1 QIDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.allocated‚607 trainer/global_step‚12089 ZLDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.freed‚ 155459309 LFDeviceStatsMonitor.on_validation_batch_start/oversize_segments.current‚0 [KDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.peak‚ 13037993984 PGDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.peak‚3677 kVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.allocated‚1114135895877120ßå3€ 4÷// _wandbruntime‚14185  _runtime‚14185…*1Õ€)ø/úR GADeviceStatsMonitor.on_validation_batch_end/oversize_segments.peak‚0 TFDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.current‚ 690184120 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.freed‚0 ^IDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.freed‚1128561207729152 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.peak‚ 362095616 [LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.current‚ 1182793728 ICDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.freed‚0 trainer/global_step‚12089 _RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.current‚23940096 93DeviceStatsMonitor.on_validation_batch_end/num_ooms‚0 TFDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.current‚ 693119488 bODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.allocated‚13760539628544 NHDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.current‚2 MEDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.current‚199 PIDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.peak‚35 bMDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.allocated‚1128561374441984 `LDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.allocated‚995888802516992 OIDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.allocated‚0 ZJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.peak‚ 12392107008 BDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚1171 MEDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.current‚564 QCDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.current‚ 693119488 VGDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.peak‚ 1177047040 MGDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.current‚0 RLDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.current‚2 ePDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.freed‚1114191321409024 D;DeviceStatsMonitor.on_validation_batch_end/num_device_alloc‚1171 ^KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.freed‚13708577509054 H>DeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚31931 KBDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.peak‚1212 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.freed‚72713044 VHDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.peak‚ 401960960 JBDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.peak‚607 YDDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.freed‚1009648649026048 D:DeviceStatsMonitor.on_validation_batch_end/active.all.peak‚31931 QCDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.freed‚ 228181793 _KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.freed‚973587023608576 MEDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.current‚607 WJDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.current‚18002944 cODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.allocated‚995888802516992 RIDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.peak‚1187 ^NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 $ _timestamp‚1.788524559806598e+09 M?DeviceStatsMonitor.on_validation_batch_end/allocation.all.freed‚ 553587874 \NDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.allocated‚ 155468751 YKDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.current‚ 166712832 gTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.allocated‚14370029092864 JDDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.peak‚0¿#q·…)ù/ÿR ]NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.allocated‚ 1182793728 I;DeviceStatsMonitor.on_validation_batch_end/active.all.freed‚ 553587874 _KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.freed‚995888784514048 TFDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.freed‚ 303565210 KEDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.freed‚0 LFDeviceStatsMonitor.on_validation_batch_end/oversize_segments.allocated‚0 ICDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.freed‚0 TFDeviceStatsMonitor.on_validation_batch_end/active.large_pool.allocated‚ 250022666 WGDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.peak‚ 12835443712 C=DeviceStatsMonitor.on_validation_batch_end/num_oom_rejections‚0 JADeviceStatsMonitor.on_validation_batch_end/active.large_pool.peak‚3677 ZMDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.current‚18002944 YJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.peak‚ 1177047040 G=DeviceStatsMonitor.on_validation_batch_end/active.all.current‚20521 HBDeviceStatsMonitor.on_validation_batch_end/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.peak‚ 14220787712 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.freed‚0 I@DeviceStatsMonitor.on_validation_batch_end/segment.all.allocated‚1171 NDDeviceStatsMonitor.on_validation_batch_end/active.small_pool.current‚20519 WGDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.allocated‚ 14220787712 ICDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.freed‚0 ZEDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.allocated‚1009649342145536 UGDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.allocated‚ 228181992 @:DeviceStatsMonitor.on_validation_batch_end/num_device_free‚0 _step‚992 JDDeviceStatsMonitor.on_validation_batch_end/active.large_pool.current‚2 XJDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.allocated‚ 250022666 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.allocated‚72713241 \LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.current‚ 13037993984 G>DeviceStatsMonitor.on_validation_batch_end/segment.all.current‚1171 MEDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.current‚564 QCDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.current‚ 693119488 VGDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.peak‚ 1177047040 MGDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.current‚0 RLDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.current‚2 ePDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.freed‚1114191321409024 D;DeviceStatsMonitor.on_validation_batch_end/num_device_alloc‚1171 ^KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.freed‚13708577509054 H>DeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚31931 KBDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.peak‚1212 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.freed‚72713044 VHDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.peak‚ 401960960 JBDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.peak‚607 YDDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.freed‚1009648649026048 D:DeviceStatsMonitor.on_validation_batch_end/active.all.peak‚31931 QCDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.freed‚ 228181793 _KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.freed‚973587023608576 MEDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.current‚607 WJDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.current‚18002944 cODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.allocated‚995888802516992 RIDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.peak‚1187 ^NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.allocated‚ 13037993984 $ _timestamp‚1.788524559806598e+09 M?DeviceStatsMonitor.on_validation_batch_end/allocation.all.freed‚ 553587874 \NDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.allocated‚ 155468751 YKDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.current‚ 166712832 gTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.allocated‚14370029092864 JDDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_validation_batch_end/oversize_segments.peak‚0 TFDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.current‚ 690184120 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.freed‚0 ^IDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.freed‚1128561207729152 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.peak‚ 362095616 [LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.current‚ 1182793728 ICDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.freed‚0 trainer/global_step‚12089 _RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.current‚23940096 93DeviceStatsMonitor.on_validation_batch_end/num_ooms‚0 TFDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.current‚ 693119488 bODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.allocated‚13760539628544 NHDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.current‚2 MEDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.current‚199 PIDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.peak‚35 bMDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.allocated‚1128561374441984 `LDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.allocated‚995888802516992 OIDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.allocated‚0 ZJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.peak‚ 12392107008 B&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 particle/accuracy_4_of_7‚1 9"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.4444444179534912 & Purity/3FRt0SRqqt2FBt/FRt_purity‚0 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.01353675127029419 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 :"Purity/*FRt2SRqqt2FBt/event_purity‚0.03999999910593033 * Purity/4FRt*SRqqt*FBt/FRt_purity‚0.125 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.2992435097694397 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 particle/accuracy_2_of_3‚1 jet/accuracy_2_of_7‚0.5 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 ("Purity/4FRt*SRqqt0FBt/SRqqt_purity‚1 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.22021688520908356 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 particle/accuracy_1_of_5‚1 +"Purity/1FRt*SRqqt3FBt/SRqqt_purity‚0.75 *"Purity/0FRt2SRqqt3FBt/SRqqt_purity‚0.5 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_5_of_7‚1 7 Purity/1FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 ,jet/accuracy_4_of_4‚0.005001259967684746 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.3928571343421936 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.16866140067577362 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9278960824012756 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 particle/accuracy_1_of_3‚1 ;"Purity/0FRt3SRqqt*FBt/event_purity‚0.026143791154026985 & Purity/3FRt4SRqqt*FBt/FRt_purity‚0 7 Purity/0FRt3SRqqt*FBt/FBt_purity‚ +^ù0.8888888955116272 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.4698960483074188 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.27070707082748413 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.20000000298023224 :"Purity/0FRt0SRqqt2FBt/event_purity‚0.33546820282936096 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 8"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.430885374546051 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.22399772703647614 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.20775862038135529 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.19766083359718323 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.5221486687660217 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.28901588916778564 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 7 Purity/*FRt3SRqqt*FBt/FBt_purity‚0.9642857313156128 8 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.28796297311782837 *"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.5 jet/accuracy_4_of_6‚0 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9940325617790222 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.20266126096248627 :"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.41996997594833374 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 :"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.44999998807907104 & Purity/1FRt*SRqqt4FBt/FRt_purity‚0 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.4845688045024872 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 particle/accuracy_4_of_5‚1 /"Purity/0FRt0SRqqt3FBt/event_purity‚0.078125 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.4285714328289032 & Purity/0FRt2SRqqt2FBt/FBt_purity‚1 8 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.13333334028720856 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9921281337738037 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 & Purity/1FRt0SRqqt4FBt/FRt_purity‚0 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.4640624523162842 9"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.2916666567325592 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.2581765651702881 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 9"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.5061728358268738 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.6140351295471191 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.4841808080673218 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 6 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.429051011800766 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9917411804199219 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.4051339626312256 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.5378274917602539 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 8 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.22499999403953552 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 6 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.986111044883728 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 epoch‚9 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5164610147476196 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.5287356376647949 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 *"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.5 ;"Purity/0FRt3SRqqt0FBt/event_purity‚0.026143791154026985 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 _runtime‚14201.130075614 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9378306865692139 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.4142521023750305 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.35337358713150024 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2916755676269531 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.22839505970478058 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 ;"Purity/*FRt3SRqqt*FBt/event_purity‚0.008064515888690948 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.4830459654331207 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 particle/accuracy_1_of_2‚1 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9916380643844604 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.2083333432674408 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_3_of_5‚1 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 $ _timestamp‚1.788524575656954e+09 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9928220510482788 :"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.43039727210998535 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8154963850975037 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.28164908289909363 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.22057506442070007 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 :"Purity/*FRt*SRqqt1FBt/event_purity‚0.40941840410232544 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 ;"Purity/*FRt3SRqqt0FBt/event_purity‚0.008064515888690948 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 particle/accuracy_1_of_6‚1 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 jet/accuracy_4_of_7‚0 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.21429996192455292 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 +jet/accuracy_3_of_4‚0.06481549143791199 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 :"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.43950656056404114 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 & Purity/1FRt3SRqqt2FBt/FRt_purity‚0 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 :"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.49441391229629517 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9996549487113953 6 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.451890766620636 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.18312031030654907 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 6 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.270304799079895 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.4430861175060272 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.3373511731624603 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.2916666567325592 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 7 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.2292637974023819 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.3071075975894928 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 *"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.5 +"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.25 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2972850799560547 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 particle/accuracy_5_of_5‚1 ) Purity/1FRt1SRqqt3FBt/FRt_purity‚0.75 ) Purity/4FRt1SRqqt0FBt/FRt_purity‚0.25 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.19319970905780792 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9882482886314392 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 jet/accuracy_6_of_7‚0 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.23758848011493683 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9912986159324646 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 +jet/accuracy_4_of_5‚0.01941884309053421 :"Purity/*FRt1SRqqt2FBt/event_purity‚0.02494879625737667 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 :"Purity/2FRt*SRqqt2FBt/event_purity‚0.03488372266292572 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 9"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.6000000238418579 -"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.5625 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6467671990394592 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.4063253700733185 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.09976761788129807 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.4575652480125427 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.03606931120157242 +jet/accuracy_2_of_2‚0.10695993900299072 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 jet/accuracy_1_of_7‚1 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9912068843841553 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 particle/accuracy_7_of_7‚1 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.4596259295940399 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 particle/accuracy_2_of_6‚1 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.4558471739292145 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 particle/accuracy_1_of_1‚1 particle/accuracy_3_of_3‚1 +"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.25 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.2916666567325592 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.3410256803035736 ,particle/sensitivity‚0.36256900429725647 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 ) Purity/*FRt1SRqqt3FBt/FRt_purity‚0.75 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 *jet/accuracy_1_of_4‚0.9258734583854675 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚1 8"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.450425922870636 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9986273646354675 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.4132823646068573 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.11082696169614792 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.05863095074892044 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 jet/accuracy_1_of_6‚1 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.40232792496681213 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 :"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.43111929297447205 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.23030903935432434 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.26760634779930115 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.14245420694351196 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5461565852165222 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9969576597213745 *jet/accuracy_1_of_3‚0.8754528164863586 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9903560280799866 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9922183752059937 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 :"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.42592594027519226 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.3333333432674408 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 ("Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.20312044024467468 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.4722222089767456 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.48009032011032104 8"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.422895610332489 particle/accuracy_3_of_6‚1 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5203239321708679 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.23045267164707184 particle/accuracy_4_of_4‚1 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 :"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.46168947219848633 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 particle/accuracy_1_of_7‚1 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5225700736045837 ;"Purity/1FRt1SRqqt1FBt/event_purity‚0.035961542278528214 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.22207796573638916 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 * Purity/0FRt0SRqqt4FBt/FBt_purity‚0.875 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.6515151858329773 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 * Purity/1FRt0SRqqt3FBt/FRt_purity‚0.375 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 particle/accuracy_1_of_4‚1 :"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.48666664958000183 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.25764715671539307 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.025160469114780426 :"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.42701491713523865 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.4062500298023224 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.23287096619606018 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.5502976179122925 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.6220238208770752 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 trainer/global_step‚12089 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9980646371841431 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.3261905014514923 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.0693688690662384 6 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.977150559425354 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.020962461829185486 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 *jet/accuracy_1_of_2‚0.7450089454650879 ("Purity/4FRt*SRqqt*FBt/SRqqt_purity‚1 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9994331002235413 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.10926299542188644 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 9"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.4566666781902313 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 4 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.2722187936306 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 *jet/accuracy_1_of_1‚0.4545123279094696 *jet/accuracy_1_of_5‚0.9501855969429016 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9941520690917969 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 +jet/accuracy_2_of_3‚0.25088390707969666 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 :"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.49075162410736084 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.22399772703647614 <"Purity/1FRt2SRqqt0FBt/event_purity‚0.0012210012646391988 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.23602427542209625 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9923497438430786 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.6025437712669373 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.24738113582134247 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.4354598820209503 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9942528009414673 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_2_of_4‚1 7 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.2666666805744171 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.29374998807907104 particle/accuracy_6_of_6‚1 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.9181286692619324 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 8 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.38235294818878174 :"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.44539710879325867 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 7 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.2935185134410858 +"Purity/1FRt1SRqqt3FBt/SRqqt_purity‚0.75 ) Purity/4FRt1SRqqt*FBt/FRt_purity‚0.25 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚1 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 )loss/val_total_loss‚38.00534658286032 :"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.43317511677742004 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.1899949461221695 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 * Purity/*FRt*SRqqt4FBt/FBt_purity‚0.875 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.026606198400259018 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt2SRqqt2FBt/event_purity‚0.0714285746216774 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.4473913311958313 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.7142857313156128 :"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.46683311462402344 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.3424794673919678 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.5208333730697632 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 6validation_average_jet_accuracy‚0.4222163259983063 7 Purity/*FRt3SRqqt1FBt/FBt_purity‚0.9629629850387573 :"Purity/*FRt0SRqqt2FBt/event_purity‚0.25642696022987366 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.1666666716337204 particle/accuracy_6_of_7‚1 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 & Purity/*FRt3SRqqt2FBt/FRt_purity‚0 *"Purity/*FRt2SRqqt3FBt/SRqqt_purity‚0.5 :"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.42927977442741394 :"Purity/0FRt*SRqqt2FBt/event_purity‚0.27370932698249817 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 :"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.46861180663108826 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 :"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.44999998807907104 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.11347517371177673 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9888389706611633 jet/accuracy_5_of_5‚0 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.06578947603702545 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.4210563898086548 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.017187315970659256 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9877815842628479 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9992784857749939 8 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.40897515416145325 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 (particle/accuracy‚0.8580988645553589 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.30057454109191895 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.2513001561164856 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 particle/accuracy_2_of_5‚1 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 *"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.5 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 9"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.4078947603702545 :"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.49004635214805603 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 ,jet/accuracy_3_of_3‚0.020856354385614395 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 6 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.214559406042099 7 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.8571428656578064 8"Purity/*FRt0SRqqt3FBt/event_purity‚0.138888880610466 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.24385148286819458 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.3819444477558136 8"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.430174320936203 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9774305820465088 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2033730000257492 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 7 Purity/0FRt3SRqqt1FBt/FBt_purity‚0.8888888955116272 ,"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.625 8"Purity/0FRt*SRqqt0FBt/event_purity‚0.328537255525589 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 9"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.4113026261329651 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 6 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.949999988079071 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.23825664818286896 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.4736842215061188 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.22640913724899292 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.22324730455875397 8"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.511164665222168 , Purity/1FRt3SRqqt1FBt/FRt_purity‚0.21875 jet/accuracy_3_of_7‚0 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.5374841690063477 ;"Purity/0FRt2SRqqt1FBt/event_purity‚0.020440252497792244 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.07560566812753677 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 *jet/accuracy_2_of_5‚0.4850102663040161 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9700000286102295 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 7 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.2227163463830948 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.4063253700733185 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.4425894021987915 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9351851940155029 ."Purity/1FRt0SRqqt3FBt/event_purity‚0.21875 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.007862419821321964 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 particle/accuracy_2_of_2‚1 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.4231608211994171 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.012752167880535126 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 +particle/specificity‚0.9497824311256409 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.3888888955116272 particle/accuracy_5_of_6‚1 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.5714285969734192 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.4204545319080353 (particle/f_score‚0.44360870122909546 8"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.458333283662796 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/3FRt1SRqqt2FBt/FRt_purity‚0.1666666716337204 * Purity/4FRt*SRqqt0FBt/FRt_purity‚0.125 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 8 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.26734232902526855 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.20000000298023224 ,"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.625 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.47898098826408386 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 particle/accuracy_4_of_6‚1 & Purity/3FRt4SRqqt0FBt/FRt_purity‚0 ;"Purity/*FRt2SRqqt0FBt/event_purity‚0.028349487110972404 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 jet/accuracy_6_of_6‚0 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 + Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9375 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 *jet/accuracy_2_of_6‚0.5064102411270142 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 7 Purity/1FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.18981324136257172 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.009542706422507763 7"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.52234947681427 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.21187643706798553 +jet/accuracy_3_of_6‚0.12820512056350708 9"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.4379085600376129 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9994331002235413 <"Purity/1FRt2SRqqt*FBt/event_purity‚0.0010582010727375746 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.24242424964904785 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 6 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.992219865322113 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.10847645252943039 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.19999997317790985 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.48714154958724976 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9894230961799622 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.6212121248245239 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 6 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.095238097012043 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.6111111044883728 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.4402778148651123 jet/accuracy_7_of_7‚0 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 & Purity/*FRt*SRqqt4FBt/FRt_purity‚0 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9961150288581848 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 :"Purity/2FRt0SRqqt2FBt/event_purity‚0.05263157933950424 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.49418604373931885 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 jet/accuracy_5_of_7‚0 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.34583544731140137 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.06286532431840897 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 +jet/accuracy_3_of_5‚0.10986942797899246 ."Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.40625 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 _step‚993 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 :"Purity/0FRt*SRqqt*FBt/event_purity‚0.43670210242271423 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9444445371627808 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 particle/accuracy_3_of_7‚1 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 9L;ûÍ"Purity/*FRt0SRqqt1FBt/event_purity‚0.6863205432891846 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.2740342915058136 7 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.2222221940755844 jet/accuracy_5_of_6‚0 *"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.5 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8888888955116272 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9977324604988098 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 , Purity/*FRt2SRqqt2FBt/FRt_purity‚0.28125 :"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.46666666865348816 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546Óª´f%|þ/ý¦ :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.48714154958724976 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9894230961799622 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.6212121248245239 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 6 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.095238097012043 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.6111111044883728 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.4402778148651123 jet/accuracy_7_of_7‚0 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 & Purity/*FRt*SRqqt4FBt/FRt_purity‚0 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9961150288581848 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 :"Purity/2FRt0SRqqt2FBt/event_purity‚0.05263157933950424 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.49418604373931885 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 jet/accuracy_5_of_7‚0 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.34583544731140137 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.06286532431840897 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 +jet/accuracy_3_of_5‚0.10986942797899246 ."Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.40625 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 _step‚993 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 :"Purity/0FRt*SRqqt*FBt/event_purity‚0.43670210242271423 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9444445371627808 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 particle/accuracy_3_of_7‚1 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6863205432891846 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.2740342915058136 7 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.2222221940755844 jet/accuracy_5_of_6‚0 *"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.5 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8888888955116272 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9977324604988098 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 , Purity/*FRt2SRqqt2FBt/FRt_purity‚0.28125 :"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.46666666865348816 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9927077293395996 :"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.42839065194129944 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9938936829566956 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.25350260734558105 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.22232632339000702 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 :"Purity/0FRt1SRqqt2FBt/event_purity‚0.05818965658545494 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.2083333432674408 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9979475140571594 particle/accuracy_3_of_4‚1 -"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.5625 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_2_of_7‚1 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.1944444477558136 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 *jet/accuracy_2_of_4‚0.3550698161125183 & Purity/*FRt0SRqqt4FBt/FRt_purity‚0 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 particle/accuracy_4_of_7‚1 9"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.4444444179534912 & Purity/3FRt0SRqqt2FBt/FRt_purity‚0 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.01353675127029419 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 :"Purity/*FRt2SRqqt2FBt/event_purity‚0.03999999910593033 * Purity/4FRt*SRqqt*FBt/FRt_purity‚0.125 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.2992435097694397 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 particle/accuracy_2_of_3‚1 jet/accuracy_2_of_7‚0.5 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 ("Purity/4FRt*SRqqt0FBt/SRqqt_purity‚1 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.22021688520908356 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 particle/accuracy_1_of_5‚1 +"Purity/1FRt*SRqqt3FBt/SRqqt_purity‚0.75 *"Purity/0FRt2SRqqt3FBt/SRqqt_purity‚0.5 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_5_of_7‚1 7 Purity/1FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 ,jet/accuracy_4_of_4‚0.005001259967684746 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.3928571343421936 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.16866140067577362 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9278960824012756 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 particle/accuracy_1_of_3‚1 ;"Purity/0FRt3SRqqt*FBt/event_purity‚0.026143791154026985 & Purity/3FRt4SRqqt*FBt/FRt_purity‚0 7 Purity/0FRt3SRqqt*FBt/FBt_purity‚0.8888888955116272 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.4698960483074188 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.27070707082748413 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.20000000298023224 :"Purity/0FRt0SRqqt2FBt/event_purity‚0.33546820282936096 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 8"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.430885374546051 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.22399772703647614 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.20775862038135529 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.19766083359718323 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.5221486687660217 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.28901588916778564 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 7 Purity/*FRt3SRqqt*FBt/FBt_purity‚0.9642857313156128 8 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.28796297311782837 *"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.5 jet/accuracy_4_of_6‚0 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9940325617790222 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.20266126096248627 :"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.41996997594833374 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 :"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.44999998807907104 & Purity/1FRt*SRqqt4FBt/FRt_purity‚0 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.4845688045024872 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 particle/accuracy_4_of_5‚1 /"Purity/0FRt0SRqqt3FBt/event_purity‚0.078125 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.4285714328289032 & Purity/0FRt2SRqqt2FBt/FBt_purity‚1 8 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.13333334028720856 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9921281337738037 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 & Purity/1FRt0SRqqt4FBt/FRt_purity‚0 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.4640624523162842 9"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.2916666567325592 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.2581765651702881 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 9"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.5061728358268738 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.6140351295471191 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.4841808080673218 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 6 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.429051011800766 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9917411804199219 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.4051339626312256 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.5378274917602539 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 8 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.22499999403953552 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 6 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.986111044883728 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 epoch‚9 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5164610147476196 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.5287356376647949 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 *"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.5 ;"Purity/0FRt3SRqqt0FBt/event_purity‚0.026143791154026985 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 _runtime‚14201.130075614 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9378306865692139 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.4142521023750305 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.35337358713150024 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2916755676269531 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.22839505970478058 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 ;"Purity/*FRt3SRqqt*FBt/event_purity‚0.008064515888690948 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.4830459654331207 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 particle/accuracy_1_of_2‚1 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9916380643844604 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.2083333432674408 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_3_of_5‚1 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 $ _timestamp‚1.788524575656954e+09 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9928220510482788 :"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.43039727210998535 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8154963850975037 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.28164908289909363 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.22057506442070007 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 :"Purity/*FRt*SRqqt1FBt/event_purity‚0.40941840410232544 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 ;"Purity/*FRt3SRqqt0FBt/event_purity‚0.008064515888690948 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 particle/accuracy_1_of_6‚1 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 jet/accuracy_4_of_7‚0 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.21429996192455292 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 +jet/accuracy_3_of_4‚0.06481549143791199 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 :"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.43950656056404114 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 & Purity/1FRt3SRqqt2FBt/FRt_purity‚0 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 :"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.49441391229629517 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9996549487113953 6 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.451890766620636 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.18312031030654907 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 6 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.270304799079895 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.4430861175060272 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.3373511731624603 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.2916666567325592 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 7 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.2292637974023819 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.3071075975894928 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 *"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.5 +"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.25 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2972850799560547 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 particle/accuracy_5_of_5‚1 ) Purity/1FRt1SRqqt3FBt/FRt_purity‚0.75 ) Purity/4FRt1SRqqt0FBt/FRt_purity‚0.25 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.19319970905780792 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9882482886314392 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 jet/accuracy_6_of_7‚0 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.23758848011493683 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9912986159324646 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 +jet/accuracy_4_of_5‚0.01941884309053421 :"Purity/*FRt1SRqqt2FBt/event_purity‚0.02494879625737667 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 :"Purity/2FRt*SRqqt2FBt/event_purity‚0.03488372266292572 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 9"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.6000000238418579 -"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.5625 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6467671990394592 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.4063253700733185 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.09976761788129807 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.4575652480125427 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.03606931120157242 +jet/accuracy_2_of_2‚0.10695993900299072 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 jet/accuracy_1_of_7‚1 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9912068843841553 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 particle/accuracy_7_of_7‚1 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.4596259295940399 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 particle/accuracy_2_of_6‚1 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.4558471739292145 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 particle/accuracy_1_of_1‚1 particle/accuracy_3_of_3‚1 +"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.25 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.2916666567325592 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.3410256803035736 ,particle/sensitivity‚0.36256900429725647 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 ) Purity/*FRt1SRqqt3FBt/FRt_purity‚0.75 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 *jet/accuracy_1_of_4‚0.9258734583854675 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚1 8"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.450425922870636 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9986273646354675 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.4132823646068573 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.11082696169614792 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.05863095074892044 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 jet/accuracy_1_of_6‚1 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.40232792496681213 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 :"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.43111929297447205 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.23030903935432434 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.26760634779930115 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.14245420694351196 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5461565852165222 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9969576597213745 *jet/accuracy_1_of_3‚0.8754528164863586 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9903560280799866 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9922183752059937 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 :"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.42592594027519226 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.3333333432674408 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 ("Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.20312044024467468 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.4722222089767456 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.48009032011032104 8"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.422895610332489 particle/accuracy_3_of_6‚1 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5203239321708679 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.23045267164707184 particle/accuracy_4_of_4‚1 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 :"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.46168947219848633 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 particle/accuracy_1_of_7‚1 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5225700736045837 ;"Purity/1FRt1SRqqt1FBt/event_purity‚0.035961542278528214 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.22207796573638916 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 * Purity/0FRt0SRqqt4FBt/FBt_purity‚0.875 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.6515151858329773 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 * Purity/1FRt0SRqqt3FBt/FRt_purity‚0.375 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 particle/accuracy_1_of_4‚1 :"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.48666664958000183 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.25764715671539307 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.025160469114780426 :"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.42701491713523865 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.4062500298023224 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.23287096619606018 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.5502976179122925 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.6220238208770752 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 trainer/global_step‚12089 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9980646371841431 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.3261905014514923 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.0693688690662384 6 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.977150559425354 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.020962461829185486 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 *jet/accuracy_1_of_2‚0.7450089454650879 ("Purity/4FRt*SRqqt*FBt/SRqqt_purity‚1 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9994331002235413 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.10926299542188644 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 9"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.4566666781902313 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 4 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.2722187936306 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 *jet/accuracy_1_of_1‚0.4545123279094696 *jet/accuracy_1_of_5‚0.9501855969429016 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9941520690917969 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 +jet/accuracy_2_of_3‚0.25088390707969666 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 :"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.49075162410736084 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.22399772703647614 <"Purity/1FRt2SRqqt0FBt/event_purity‚0.0012210012646391988 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.23602427542209625 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9923497438430786 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.6025437712669373 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.24738113582134247 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.4354598820209503 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9942528009414673 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_2_of_4‚1 7 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.2666666805744171 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.29374998807907104 particle/accuracy_6_of_6‚1 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.9181286692619324 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 8 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.38235294818878174 :"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.44539710879325867 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 7 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.2935185134410858 +"Purity/1FRt1SRqqt3FBt/SRqqt_purity‚0.75 ) Purity/4FRt1SRqqt*FBt/FRt_purity‚0.25 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚1 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 )loss/val_total_loss‚38.00534658286032 :"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.43317511677742004 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.1899949461221695 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 * Purity/*FRt*SRqqt4FBt/FBt_purity‚0.875 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.026606198400259018 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt2SRqqt2FBt/event_purity‚0.0714285746216774 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.4473913311958313 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.7142857313156128 :"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.46683311462402344 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.3424794673919678 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.5208333730697632 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 6validation_average_jet_accuracy‚0.4222163259983063 7 Purity/*FRt3SRqqt1FBt/FBt_purity‚0.9629629850387573 :"Purity/*FRt0SRqqt2FBt/event_purity‚0.25642696022987366 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.1666666716337204 particle/accuracy_6_of_7‚1 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 & Purity/*FRt3SRqqt2FBt/FRt_purity‚0 *"Purity/*FRt2SRqqt3FBt/SRqqt_purity‚0.5 :"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.42927977442741394 :"Purity/0FRt*SRqqt2FBt/event_purity‚0.27370932698249817 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 :"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.46861180663108826 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 :"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.44999998807907104 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.11347517371177673 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9888389706611633 jet/accuracy_5_of_5‚0 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.06578947603702545 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.4210563898086548 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.017187315970659256 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9877815842628479 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9992784857749939 8 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.40897515416145325 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 (particle/accuracy‚0.8580988645553589 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.30057454109191895 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.2513001561164856 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 particle/accuracy_2_of_5‚1 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 *"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.5 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 9"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.4078947603702545 :"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.4900463521480560UJ+_3 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 ,jet/accuracy_3_of_3‚0.020856354385614395 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 6 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.214559406042099 7 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.8571428656578064 8"Purity/*FRt0SRqqt3FBt/event_purity‚0.138888880610466 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.24385148286819458 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.3819444477558136 8"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.430174320936203 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9774305820465088 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2033730000257492 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 7 Purity/0FRt3SRqqt1FBt/FBt_purity‚0.8888888955116272 ,"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.625 8"Purity/0FRt*SRqqt0FBt/event_purity‚0.328537255525589 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 9"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.4113026261329651 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 6 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.949999988079071 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.23825664818286896 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.4736842215061188 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.22640913724899292 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.22324730455875397 8"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.511164665222168 , Purity/1FRt3SRqqt1FBt/FRt_purity‚0.21875 jet/accuracy_3_of_7‚0 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.5374841690063477 ;"Purity/0FRt2SRqqt1FBt/event_purity‚0.020440252497792244 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.07560566812753677 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 *jet/accuracy_2_of_5‚0.4850102663040161 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9700000286102295 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 7 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.2227163463830948 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.4063253700733185 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.4425894021987915 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9351851940155029 ."Purity/1FRt0SRqqt3FBt/event_purity‚0.21875 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.007862419821321964 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 particle/accuracy_2_of_2‚1 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.4231608211994171 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.012752167880535126 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 +particle/specificity‚0.9497824311256409 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.3888888955116272 particle/accuracy_5_of_6‚1 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.5714285969734192 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.4204545319080353 (particle/f_score‚0.44360870122909546 8"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.458333283662796 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/3FRt1SRqqt2FBt/FRt_purity‚0.1666666716337204 * Purity/4FRt*SRqqt0FBt/FRt_purity‚0.125 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 8 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.26734232902526855 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.20000000298023224 ,"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.625 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.47898098826408386 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 particle/accuracy_4_of_6‚1 & Purity/3FRt4SRqqt0FBt/FRt_purity‚0 ;"Purity/*FRt2SRqqt0FBt/event_purity‚0.028349487110972404 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 jet/accuracy_6_of_6‚0 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 + Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9375 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 *jet/accuracy_2_of_6‚0.5064102411270142 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 7 Purity/1FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.18981324136257172 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.009542706422507763 7"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.52234947681427 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.21187643706798553 +jet/accuracy_3_of_6‚0.12820512056350708 9"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.4379085600376129 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9994331002235413 <"Purity/1FRt2SRqqt*FBt/event_purity‚0.0010582010727375746 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.24242424964904785 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 6 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.992219865322113 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.10847645252943039 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.19999997317790985áÁ,.°Rÿ/ n20ypkh2j@ ŸðêÔЫã0`Trainer.fit` stopped: `max_epochs=10` reached. ÕA:Éž€0 n20ypkh2j‹ ¡ðêÔÈÜä3zEpoch 9/9 â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â” 1209/1209 0:23:03 • 0:00:00 0.94it/s v_num: pkh2kbw$0 n20ypkh2j ¡ðêÔ¸®ƒ4 ;¶x4‚0/ _wandbruntime‚14204  _runtime‚14204ÝtûGƒ0‚(’ün